Are we properly prepared to handle 3 Socket setups?

Are we properly prepared to handle 3 Socket setups?

[Jesper Juhl]

Hi,

This may be a little off topic, but I think it's interresting enough
to warrent a single mail.

I just saw a news article (http://www.theinquirer.net/?article=41610)
about a 3 Socket Opteron motherboard and couldn't help but wonder if
we are prepared to deal with such a beast, so I thought I'd inform
everyone :-)

I'm guessing equipping such a board with 3 single core CPU's could
show up some interresting corner cases in schedular code and
elsewhere, I'll bet we have some assumptions somewhere about
nr_of_cpus being an even number... In any case it would be an
interresting box to test on, so for those of you employed by big
business with the cash to purchase a test box, it would at least be an
interresting one to add to the mix :)

Anyway, just wanted to give everyone a heads-up, bye bye...

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Are we properly prepared to handle 3 Socket setups?

[Rene Herman]

On 08/12/2007 03:08 AM, Jesper Juhl wrote:

> This may be a little off topic, but I think it's interresting enough
> to warrent a single mail.
> 
> I just saw a news article (http://www.theinquirer.net/?article=41610)
> about a 3 Socket Opteron motherboard and couldn't help but wonder if
> we are prepared to deal with such a beast, so I thought I'd inform
> everyone :-)
> 
> I'm guessing equipping such a board with 3 single core CPU's could
> show up some interresting corner cases in schedular code and
> elsewhere, I'll bet we have some assumptions somewhere about
> nr_of_cpus being an even number...

I would hope the N=1 case will have flushed out enough of those... :-|

Rene.

Re: Are we properly prepared to handle 3 Socket setups?

[Jesper Juhl]

On 12/08/07, Rene Herman <rene.herman@gmail.com> wrote:
> On 08/12/2007 03:08 AM, Jesper Juhl wrote:
>
> > This may be a little off topic, but I think it's interresting enough
> > to warrent a single mail.
> >
> > I just saw a news article (http://www.theinquirer.net/?article=41610)
> > about a 3 Socket Opteron motherboard and couldn't help but wonder if
> > we are prepared to deal with such a beast, so I thought I'd inform
> > everyone :-)
> >
> > I'm guessing equipping such a board with 3 single core CPU's could
> > show up some interresting corner cases in schedular code and
> > elsewhere, I'll bet we have some assumptions somewhere about
> > nr_of_cpus being an even number...
>
> I would hope the N=1 case will have flushed out enough of those... :-|
>
Hehe, true, but I was thinking more of nr_of_cpus is an odd number > 1. :-)
Just thinking of having to divide things by 3 makes me worry ;-)  ...

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Are we properly prepared to handle 3 Socket setups?

[Rene Herman]

On 08/12/2007 03:52 AM, Jesper Juhl wrote:

> On 12/08/07, Rene Herman <rene.herman@gmail.com> wrote:
>> On 08/12/2007 03:08 AM, Jesper Juhl wrote:
>>
>>> This may be a little off topic, but I think it's interresting enough
>>> to warrent a single mail.
>>>
>>> I just saw a news article (http://www.theinquirer.net/?article=41610)
>>> about a 3 Socket Opteron motherboard and couldn't help but wonder if
>>> we are prepared to deal with such a beast, so I thought I'd inform
>>> everyone :-)
>>>
>>> I'm guessing equipping such a board with 3 single core CPU's could
>>> show up some interresting corner cases in schedular code and
>>> elsewhere, I'll bet we have some assumptions somewhere about
>>> nr_of_cpus being an even number...
>> I would hope the N=1 case will have flushed out enough of those... :-|
>>
> Hehe, true, but I was thinking more of nr_of_cpus is an odd number > 1. :-)
> Just thinking of having to divide things by 3 makes me worry ;-)  ...

It's not a hugely strange worry no. Grepping around (for num_online_cpus for 
example) didn't throw up any glaring bugs I believe.

A possible problem in mm/vmstat.c:calculate_threshold() where 3 CPUs would 
be treated as 2 through an fls(). No idea about that code and if that would 
be a problem.

The line just below where it does that _does_ seem to have a problem:

         /*
          * Maximum threshold is 125
          */
         threshold = min(125, threshold);

as either the comment or the code is wrong and it seems it's the code. Added 
Andrew Morton to the CC for that.

CFS (v19.1) has an ilog2 on num_online_cpus() in 
kernel/sched.c:sched_init_granularity() but this seems not a problem. Added 
Ingo Molnar.

Rene.

Re: Are we properly prepared to handle 3 Socket setups?

[Roland Dreier]

 >         /*
 >          * Maximum threshold is 125
 >          */
 >         threshold = min(125, threshold);
 > 
 > as either the comment or the code is wrong and it seems it's the
 > code.

What's the problem?  That line sets threshold to the smaller of the
current value or 125, which is exactly what one would want to do if
the maximum value is 125.  Just do a couple of examples: eg if threshold
is 100 going into that line, then the value is left alone; if
threshold is 150 then it gets set to 125; and that seems exactly correct.

 - R.

Re: Are we properly prepared to handle 3 Socket setups?

[Rene Herman]

On 08/12/2007 05:29 AM, Roland Dreier wrote:

>  >         /*
>  >          * Maximum threshold is 125
>  >          */
>  >         threshold = min(125, threshold);
>  > 
>  > as either the comment or the code is wrong and it seems it's the
>  > code.
> 
> What's the problem?  That line sets threshold to the smaller of the
> current value or 125, which is exactly what one would want to do if
> the maximum value is 125.  Just do a couple of examples: eg if threshold
> is 100 going into that line, then the value is left alone; if
> threshold is 150 then it gets set to 125; and that seems exactly correct.

Crap, need bed, sorry (the fls/ilog2 things still stand -- quite possibly 
both not a problem either).

Rene.

Re: Are we properly prepared to handle 3 Socket setups?

[Andrew Morton]

On Sun, 12 Aug 2007 05:17:10 +0200 Rene Herman <rene.herman@gmail.com> wrote:

> On 08/12/2007 03:52 AM, Jesper Juhl wrote:
> 
> > On 12/08/07, Rene Herman <rene.herman@gmail.com> wrote:
> >> On 08/12/2007 03:08 AM, Jesper Juhl wrote:
> >>
> >>> This may be a little off topic, but I think it's interresting enough
> >>> to warrent a single mail.
> >>>
> >>> I just saw a news article (http://www.theinquirer.net/?article=41610)
> >>> about a 3 Socket Opteron motherboard and couldn't help but wonder if
> >>> we are prepared to deal with such a beast, so I thought I'd inform
> >>> everyone :-)
> >>>
> >>> I'm guessing equipping such a board with 3 single core CPU's could
> >>> show up some interresting corner cases in schedular code and
> >>> elsewhere, I'll bet we have some assumptions somewhere about
> >>> nr_of_cpus being an even number...
> >> I would hope the N=1 case will have flushed out enough of those... :-|
> >>
> > Hehe, true, but I was thinking more of nr_of_cpus is an odd number > 1. :-)
> > Just thinking of having to divide things by 3 makes me worry ;-)  ...
> 
> It's not a hugely strange worry no. Grepping around (for num_online_cpus for 
> example) didn't throw up any glaring bugs I believe.
> 
> A possible problem in mm/vmstat.c:calculate_threshold() where 3 CPUs would 
> be treated as 2 through an fls(). No idea about that code and if that would 
> be a problem.

No, that'll be OK.

> The line just below where it does that _does_ seem to have a problem:
> 
>          /*
>           * Maximum threshold is 125
>           */
>          threshold = min(125, threshold);
> 
> as either the comment or the code is wrong and it seems it's the code. Added 
> Andrew Morton to the CC for that.

Yes, that's inconsistent.  And looking at Christoph's df9ecaba it's unclear
whether the comment is wrong or the code is wrong.  The code is wrong, I
expect.

Re: Are we properly prepared to handle 3 Socket setups?

[Rene Herman]

On 08/12/2007 10:35 AM, Andrew Morton wrote:

> On Sun, 12 Aug 2007 05:17:10 +0200 Rene Herman <rene.herman@gmail.com> wrote:

>> The line just below where it does that _does_ seem to have a problem:
>>
>>          /*
>>           * Maximum threshold is 125
>>           */
>>          threshold = min(125, threshold);
>>
>> as either the comment or the code is wrong and it seems it's the code. Added 
>> Andrew Morton to the CC for that.
> 
> Yes, that's inconsistent.  And looking at Christoph's df9ecaba it's unclear
> whether the comment is wrong or the code is wrong.  The code is wrong, I
> expect.

Extremely friendly of you to pretend I wasn't being thick at all but don't 
worry, I can take it. Anyways, since Christoph wasn't in CC on that one:

On 08/12/2007 05:29 AM, Roland Dreier wrote:

> What's the problem? That line sets threshold to the smaller of the 
> current value or 125, which is exactly what one would want to do if the
> maximum value is 125. Just do a couple of examples: eg if threshold is
> 100 going into that line, then the value is left alone; if threshold is
> 150 then it gets set to 125; and that seems exactly correct.

Rene.

Re: Are we properly prepared to handle 3 Socket setups?

[Christoph Lameter]

On Sun, 12 Aug 2007, Rene Herman wrote:

> > >          /*
> > >           * Maximum threshold is 125
> > >           */
> > >          threshold = min(125, threshold);
> > > 
> > > as either the comment or the code is wrong and it seems it's the code.
> > > Added Andrew Morton to the CC for that.
> > 
> > Yes, that's inconsistent.  And looking at Christoph's df9ecaba it's unclear
> > whether the comment is wrong or the code is wrong.  The code is wrong, I
> > expect.

?? What is wrong with the code? We want the threshold to stay below 125?
 
> On 08/12/2007 05:29 AM, Roland Dreier wrote:
> 
> > What's the problem? That line sets threshold to the smaller of the current
> > value or 125, which is exactly what one would want to do if the
> > maximum value is 125. Just do a couple of examples: eg if threshold is
> > 100 going into that line, then the value is left alone; if threshold is
> > 150 then it gets set to 125; and that seems exactly correct.

Exactly.

Re: Are we properly prepared to handle 3 Socket setups?

[Paul Mundt]

On Sun, Aug 12, 2007 at 03:52:48AM +0200, Jesper Juhl wrote:
> On 12/08/07, Rene Herman <rene.herman@gmail.com> wrote:
> > On 08/12/2007 03:08 AM, Jesper Juhl wrote:
> >
> > > This may be a little off topic, but I think it's interresting enough
> > > to warrent a single mail.
> > >
> > > I just saw a news article (http://www.theinquirer.net/?article=41610)
> > > about a 3 Socket Opteron motherboard and couldn't help but wonder if
> > > we are prepared to deal with such a beast, so I thought I'd inform
> > > everyone :-)
> > >
> > > I'm guessing equipping such a board with 3 single core CPU's could
> > > show up some interresting corner cases in schedular code and
> > > elsewhere, I'll bet we have some assumptions somewhere about
> > > nr_of_cpus being an even number...
> >
> > I would hope the N=1 case will have flushed out enough of those... :-|
> >
> Hehe, true, but I was thinking more of nr_of_cpus is an odd number > 1. :-)
> Just thinking of having to divide things by 3 makes me worry ;-)  ...
> 
These sorts of cases happen frequently enough today as it is in an
operational capacity, CPU hotplug will have the sytem in such states
already on systems with an even number. This is generally why things like
the online map and corresponding cpumasks are what counts, while the
total number of possible CPUs can be a drastically different number.

Re: Are we properly prepared to handle 3 Socket setups?

[Willy Tarreau]

On Sun, Aug 12, 2007 at 03:27:58AM +0200, Rene Herman wrote:
> On 08/12/2007 03:08 AM, Jesper Juhl wrote:
> 
> >This may be a little off topic, but I think it's interresting enough
> >to warrent a single mail.
> >
> >I just saw a news article (http://www.theinquirer.net/?article=41610)
> >about a 3 Socket Opteron motherboard and couldn't help but wonder if
> >we are prepared to deal with such a beast, so I thought I'd inform
> >everyone :-)
> >
> >I'm guessing equipping such a board with 3 single core CPU's could
> >show up some interresting corner cases in schedular code and
> >elsewhere, I'll bet we have some assumptions somewhere about
> >nr_of_cpus being an even number...
> 
> I would hope the N=1 case will have flushed out enough of those... :-|

Well, 1 could be seen as even number in base 2, it's 2^0 and has only
one bit set.

Willy

Re: Are we properly prepared to handle 3 Socket setups?

[Andi Kleen]

"Jesper Juhl" <jesper.juhl@gmail.com> writes:

> I just saw a news article (http://www.theinquirer.net/?article=41610)
> about a 3 Socket Opteron motherboard and couldn't help but wonder if
> we are prepared to deal with such a beast, so I thought I'd inform
> everyone :-)

It should work. Non power of two cores per socket used to be problematic
though, but that has been also fixed.

> nr_of_cpus being an even number... In any case it would be an
> interresting box to test on, so for those of you employed by big
> business with the cash to purchase a test box, it would at least be an
> interresting one to add to the mix :)

You can already test it by either using qemu/xen/etc. or by 
off lining one of your cores if you have more than 2.

-Andi