Scheduling with HyperThreading

Scheduling with HyperThreading

[Mike Sullivan]

What kernel versions will attempt to distribute jobs across physical CPUs on
Xeon SMP configurations.

                                                                        
                        Mike

-- 
----------------------------------------------------------------------
Mike Sullivan                           Director Performance Computing
@lliance Technologies,                  Voice: (416) 385-3255, 
18 Wynford Dr, Suite 407                Fax:   (416) 385-1774
Toronto, ON, Canada, M3C-3S2            Toll Free:1-877-216-3199
http://www.alltec.com

Re: Scheduling with HyperThreading

[James Bourne]

On Mon, 24 Feb 2003, Mike Sullivan wrote:

> What kernel versions will attempt to distribute jobs across physical CPUs on
> Xeon SMP configurations.

>From what I've heard, Arjans' user space daemon might be the way
things are going, it's at http://people.redhat.com/arjanv/irqbalance/ .

The other way that you might try is the irq load balance patch that Ingo
produced.  There is a patch that is from 2.4.20 at
http://www.hardrock.org/kernel/2.4.20/ and it is what I'm using at work on
our current Xeon systems (until I have the chance to test the user space
daemon at least).

Hope that helps.

Regards
James Bourne

>                         Mike

-- 
James Bourne, Supervisor Data Centre Operations
Mount Royal College, Calgary, AB, CA
www.mtroyal.ab.ca

"There are only 10 types of people in this world: those who
understand binary and those who don't."

Re: Scheduling with HyperThreading

[William Lee Irwin III]

On Mon, 24 Feb 2003, Mike Sullivan wrote:
>> What kernel versions will attempt to distribute jobs across physical CPUs on
>> Xeon SMP configurations.

On Mon, Feb 24, 2003 at 10:45:18PM -0700, James Bourne wrote:
> From what I've heard, Arjans' user space daemon might be the way
> things are going, it's at http://people.redhat.com/arjanv/irqbalance/ .
> The other way that you might try is the irq load balance patch that Ingo
> produced.  There is a patch that is from 2.4.20 at
> http://www.hardrock.org/kernel/2.4.20/ and it is what I'm using at work on
> our current Xeon systems (until I have the chance to test the user space
> daemon at least).

I think he's referring to the cpu scheduler, not interrupt load
balancing. mingo might have some insight into current patches for
this and current results thereof. I don't really participate in
the scheduler aside from very occasional bugfixing.


-- wli

Scheduling with Hyperthreading

[Mike Sullivan]

Sorry, should have been more specific. I am referring to the scheduler. 
I have noticed
that if I run two compute intensive jobs on a Dual Xeon, the processes 
run on separate
physical cpus and can spend a significant amount of time with both on a 
single
cpu. I have seen some patches that try to improve on this and was 
wondering if
they have made it into the production kernel stream. I ran tests on a 
2.4.20 kernel and
it had the same problem as a 2.4.18 kernel.


                                                                        
                                    Thanks
                                                                        
                                     Mike

William Lee Irwin III wrote:

>On Mon, 24 Feb 2003, Mike Sullivan wrote:
>  
>
>>>What kernel versions will attempt to distribute jobs across physical CPUs on
>>>Xeon SMP configurations.
>>>      
>>>
>
>On Mon, Feb 24, 2003 at 10:45:18PM -0700, James Bourne wrote:
>  
>
>>From what I've heard, Arjans' user space daemon might be the way
>>things are going, it's at http://people.redhat.com/arjanv/irqbalance/ .
>>The other way that you might try is the irq load balance patch that Ingo
>>produced.  There is a patch that is from 2.4.20 at
>>http://www.hardrock.org/kernel/2.4.20/ and it is what I'm using at work on
>>our current Xeon systems (until I have the chance to test the user space
>>daemon at least).
>>    
>>
>
>I think he's referring to the cpu scheduler, not interrupt load
>balancing. mingo might have some insight into current patches for
>this and current results thereof. I don't really participate in
>the scheduler aside from very occasional bugfixing.
>
>
>-- wli
>  
>



-- 
----------------------------------------------------------------------
Mike Sullivan                           Director Performance Computing
@lliance Technologies,                  Voice: (416) 385-3255,
18 Wynford Dr, Suite 407                Fax:   (416) 385-1774
Toronto, ON, Canada, M3C-3S2            Toll Free:1-877-216-3199
http://www.alltec.com

Re: Scheduling with Hyperthreading

[Mike Sullivan]

  Mark

I would to a quick snap with top, and when I saw 99.9% I assumed the the 
process had
been there during the time top was starting up.

Looking at /proc/(pid)/cpu, shows that with two jobs running they are 
sticking to cpu 0 and 1
which are siblings


                                                                        
                                                                        
Regards
                                                                        
                                                                        Mike

Mark Hahn wrote:

>>that if I run two compute intensive jobs on a Dual Xeon, the processes 
>>run on separate
>>physical cpus and can spend a significant amount of time with both on a 
>>single
>>cpu.
>>    
>>
>
>how did you determine this?  running another program, such as top,
>will naturally disturb the scheduler and corrupt any observations.
>the only means I can think of is to look in /proc/<pid>/cpu near
>very infrequently (ideally, just before the processes exit.)
>or is this what you've done?
>  
>

-- 
----------------------------------------------------------------------
Mike Sullivan                           Director Performance Computing
@lliance Technologies,                  Voice: (416) 385-3255, 
18 Wynford Dr, Suite 407                Fax:   (416) 385-1774
Toronto, ON, Canada, M3C-3S2            Toll Free:1-877-216-3199
http://www.alltec.com

RE: Scheduling with Hyperthreading

[Pallipadi, Venkatesh]

What is the nature of this jobs that you are running?
1) Just the continuous computations from the user land
2) computations mixed with some system calls, i/o or some other of sleep
wakeups.

My guess at this moment is that it is the former one. With that kind of
a load, one can easily get stuck in this "bad" situation with 2 jobs
running on the same package. And as these processes doesn't sleep or get
scheduled out, they continue to run on the same CPUs, mostly trying to
use any cache affinity in those CPUs.

2.4.20 has code in place to take care of load as in 2 above. When trying
to find the best CPU to schedule in a process (while waking up from
sleep), it actually looks at idle package v/s idle logical processor.


Thanks,
-Venkatesh

> -----Original Message-----
> From: Mike Sullivan [mailto:mike.sullivan@alltec.com] 
> Sent: Tuesday, February 25, 2003 1:01 PM
> To: Mark Hahn; linux-kernel
> Subject: Re: Scheduling with Hyperthreading
> 
> 
>   Mark
> 
> I would to a quick snap with top, and when I saw 99.9% I 
> assumed the the 
> process had
> been there during the time top was starting up.
> 
> Looking at /proc/(pid)/cpu, shows that with two jobs running they are 
> sticking to cpu 0 and 1
> which are siblings
> 
> 
>                                                               
>           
>                                                               
>           
> Regards
>                                                               
>           
>                                                               
>           Mike
> 
> Mark Hahn wrote:
> 
> >>that if I run two compute intensive jobs on a Dual Xeon, 
> the processes 
> >>run on separate
> >>physical cpus and can spend a significant amount of time 
> with both on a 
> >>single
> >>cpu.
> >>    
> >>
> >
> >how did you determine this?  running another program, such as top,
> >will naturally disturb the scheduler and corrupt any observations.
> >the only means I can think of is to look in /proc/<pid>/cpu near
> >very infrequently (ideally, just before the processes exit.)
> >or is this what you've done?
> >  
> >
> 
> -- 
> ----------------------------------------------------------------------
> Mike Sullivan                           Director Performance Computing
> @lliance Technologies,                  Voice: (416) 385-3255, 
> 18 Wynford Dr, Suite 407                Fax:   (416) 385-1774
> Toronto, ON, Canada, M3C-3S2            Toll Free:1-877-216-3199
> http://www.alltec.com
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Scheduling with Hyperthreading

[Mike Sullivan]

I have tried RH 2.4.18 stock redhat kernels and and 2.4.20 kernels. Both 
seem to
do badly.

Is there somewhere to look to see a history of the schedular work, or do 
I need
to puruse all of the changelogs to get an idea of what kernel I should 
be trying.



                                                                        
                                            Regards
                                                                        
                                            Mike

Mark Hahn wrote:

>>I would to a quick snap with top, and when I saw 99.9% I assumed the the 
>>process had
>>been there during the time top was starting up.
>>
>>Looking at /proc/(pid)/cpu, shows that with two jobs running they are 
>>sticking to cpu 0 and 1
>>which are siblings
>>    
>>
>
>ah, sorry if you said this, but which kernel are you running?
>you need a HT-aware scheduler, for sure.  does your problem go away
>if you boot with noht?
>  
>

-- 
----------------------------------------------------------------------
Mike Sullivan                           Director Performance Computing
@lliance Technologies,                  Voice: (416) 385-3255, 
18 Wynford Dr, Suite 407                Fax:   (416) 385-1774
Toronto, ON, Canada, M3C-3S2            Toll Free:1-877-216-3199
http://www.alltec.com