[PATCH 0/10] bulk cpu removal support

[PATCH 0/10] bulk cpu removal support

[Shaohua Li]

CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
To remove multiple CPUs, we should iteratively do single cpu removal.
If tasks and interrupts are migrated to a cpu which will be soon
removed, then we will trash tasks and interrupts again. The following
patches allow remove several cpus one time. It's fast and avoids
unnecessary repeated trash tasks and interrupts. This will help NUMA
style hardware removal and SMP suspend/resume. Comments and suggestions
are appreciated.

Thanks,
Shaohua

Re: [PATCH 0/10] bulk cpu removal support

[Nathan Lynch]

Shaohua Li wrote:
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.

Some quantification of the benefits of adding this complexity would be
appreciated.  Like, how long does it take to offline all the cpus in a
node serially, and how much faster is the bulk method?

Re: [PATCH 0/10] bulk cpu removal support

[Shaohua Li]

On Mon, 2006-05-08 at 01:29 -0500, Nathan Lynch wrote:
> Shaohua Li wrote:
> > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> > To remove multiple CPUs, we should iteratively do single cpu removal.
> > If tasks and interrupts are migrated to a cpu which will be soon
> > removed, then we will trash tasks and interrupts again. The following
> > patches allow remove several cpus one time. It's fast and avoids
> > unnecessary repeated trash tasks and interrupts. This will help NUMA
> > style hardware removal and SMP suspend/resume. Comments and suggestions
> > are appreciated.
> 
> Some quantification of the benefits of adding this complexity would be
> appreciated.  Like, how long does it take to offline all the cpus in a
> node serially, and how much faster is the bulk method?
This depends on the workload and I haven't a data at hand, sorry. Fast
(less tasks/timers/softirqs migration) is one benefit. The method also
means less time in stop_machine_run and less interrupt redirection, so
smaller impact to the whole system.

Thanks,
Shaohua

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Mon, May 08, 2006 at 01:45:37PM +0800, Shaohua Li wrote:
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.
> 

Thanks shaohua, since the initial code has been collecting dust for 
some several months now, and I never got time to finish it.

Think you forgot to cc the ia64 crowd. We tried to do it a little bit more
generic way, but some required arch assist due to some differences.

Andrew: please help stage in -mm since we would need more extensive testing
and help feedback to fix any before becomming prime time.

It would also help to check if other archs have any compile issues etc..

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [PATCH 0/10] bulk cpu removal support

[Andrew Morton]

Shaohua Li <shaohua.li@intel.com> wrote:
>
> CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> To remove multiple CPUs, we should iteratively do single cpu removal.
> If tasks and interrupts are migrated to a cpu which will be soon
> removed, then we will trash tasks and interrupts again. The following
> patches allow remove several cpus one time. It's fast and avoids
> unnecessary repeated trash tasks and interrupts. This will help NUMA
> style hardware removal and SMP suspend/resume. Comments and suggestions
> are appreciated.

This seems an awful lot of code for something which happens so infrequently.

How big is the problem you're fixing here, and what are the
user-observeable effects of these changes?

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
Hi Andrew,

> Shaohua Li <shaohua.li@intel.com> wrote:
> >
> > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> 
> This seems an awful lot of code for something which happens so infrequently.
> 
> How big is the problem you're fixing here, and what are the
> user-observeable effects of these changes?

This is useful when say a NUMA node is being removed. With new multi-core
CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
per socket. On NUMA node with 4 sockets, a node removal will mean we 
do 16 single cpu offlines. Each time the process and interrupts could
end up on a CPU that might be removed just immediatly.

The same is also useful for SMP Suspend/resume cases since the logical offline
is same here as well.

Even thought the code changes seem a lot, most of it is just preparation of
functions ready to accept a cpumask_t instead of a single cpu like earlier.
The reason we split them to smaller chunks so the scope of change is well
understood with each patch.

The major changes are

 - stop machine to run cpu offline functions on each cpu going offline
 - prepare offline functions in offline path to take cpumask_t
 - Some task migrate dead lock removal consideration that we ran into
    during stress test.

I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386 
and x86_64.

once we get some time deltas on a bigger machine it will help a lot. 
Iam also trying to check with some OEM';s who have such large machines for
some data.. keep posted.

ashokr

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [PATCH 0/10] bulk cpu removal support

[Andrew Morton]

Ashok Raj <ashok.raj@intel.com> wrote:
>
> On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
> Hi Andrew,
> 
> > Shaohua Li <shaohua.li@intel.com> wrote:
> > >
> > > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> > 
> > This seems an awful lot of code for something which happens so infrequently.
> > 
> > How big is the problem you're fixing here, and what are the
> > user-observeable effects of these changes?
> 
> This is useful when say a NUMA node is being removed. With new multi-core
> CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
> per socket. On NUMA node with 4 sockets, a node removal will mean we 
> do 16 single cpu offlines. Each time the process and interrupts could
> end up on a CPU that might be removed just immediatly.
> 
> The same is also useful for SMP Suspend/resume cases since the logical offline
> is same here as well.
> 
> Even thought the code changes seem a lot, most of it is just preparation of
> functions ready to accept a cpumask_t instead of a single cpu like earlier.
> The reason we split them to smaller chunks so the scope of change is well
> understood with each patch.
> 
> The major changes are
> 
>  - stop machine to run cpu offline functions on each cpu going offline
>  - prepare offline functions in offline path to take cpumask_t
>  - Some task migrate dead lock removal consideration that we ran into
>     during stress test.
> 
> I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386 
> and x86_64.
> 
> once we get some time deltas on a bigger machine it will help a lot. 
> Iam also trying to check with some OEM';s who have such large machines for
> some data.. keep posted.
> 

OK, thanks.  I'm a little surprised that this patch wasn't accompanied by a
problem description, really.  I mean, if a single CPU offlining takes three
milliseconds then why bother?

I assume it must take much longer, else you wouldn't have written the code.
Have you any ballpark numbers for how long it _does_ take?

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Thu, May 11, 2006 at 10:02:15AM -0700, Andrew Morton wrote:
> 
> OK, thanks.  I'm a little surprised that this patch wasn't accompanied by a
> problem description, really.  I mean, if a single CPU offlining takes three
> milliseconds then why bother?

It depends on whats running at the time... with some light load, i measured 
wall clock time, i remember seeing 2 secs at times, but its been a long time
i did that.. so take that with a pinch :-)_

i will try to get those idle and load times worked out again... the best
i have is a  16 way, if i get help from big system oems i will send the 
numbers out
> 
> I assume it must take much longer, else you wouldn't have written the code.
> Have you any ballpark numbers for how long it _does_ take?

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [PATCH 0/10] bulk cpu removal support

[Martin Bligh]

Ashok Raj wrote:
> On Thu, May 11, 2006 at 10:02:15AM -0700, Andrew Morton wrote:
> 
>>OK, thanks.  I'm a little surprised that this patch wasn't accompanied by a
>>problem description, really.  I mean, if a single CPU offlining takes three
>>milliseconds then why bother?
> 
> 
> It depends on whats running at the time... with some light load, i measured 
> wall clock time, i remember seeing 2 secs at times, but its been a long time
> i did that.. so take that with a pinch :-)_
> 
> i will try to get those idle and load times worked out again... the best
> i have is a  16 way, if i get help from big system oems i will send the 
> numbers out

Why is taking 30s to offline CPUs a problem?

M.

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> Ashok Raj wrote:
> > 
> > 
> > It depends on whats running at the time... with some light load, i measured 
> > wall clock time, i remember seeing 2 secs at times, but its been a long time
> > i did that.. so take that with a pinch :-)_
> > 
> > i will try to get those idle and load times worked out again... the best
> > i have is a  16 way, if i get help from big system oems i will send the 
> > numbers out
> 
> Why is taking 30s to offline CPUs a problem?
> 

Well, the real problem is that for each cpu offline we schedule a RT thread
kstopmachine() on each cpu, then turn off interrupts until this one cpu has 
removed. stop_machine_run() is a big enough sledge hammer during cpu offline
and doing this repeatedly... say on a 4 socket system, where each socket=16
logical cpus.

the system would tend to get hick ups 64 times, since we do the stopmachine
thread once for each logical cpu. When we want to replace a node for
reliability reasons, its not clear if this hick ups is a good thing.

Doing kstopmachine() on a single system is in itself noticable, what we heard
from some OEM's is this would have other app level impact as well.

With the bulk removal, we do stop machine just once, but all the 16 cpus
get removed once hence there is just one hickup, instead of 64.

Less time to offline, avoid process and interrupt bouncing on and off a cpu
which is just about to be offlined are almost extra fringe benefits you get 
with the bulk removal approach.

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [PATCH 0/10] bulk cpu removal support

[Nathan Lynch]

Ashok Raj wrote:
> On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> > Ashok Raj wrote:
> > > 
> > > 
> > > It depends on whats running at the time... with some light load, i measured 
> > > wall clock time, i remember seeing 2 secs at times, but its been a long time
> > > i did that.. so take that with a pinch :-)_
> > > 
> > > i will try to get those idle and load times worked out again... the best
> > > i have is a  16 way, if i get help from big system oems i will send the 
> > > numbers out
> > 
> > Why is taking 30s to offline CPUs a problem?
> > 
> 
> Well, the real problem is that for each cpu offline we schedule a RT thread
> kstopmachine() on each cpu, then turn off interrupts until this one cpu has 
> removed. stop_machine_run() is a big enough sledge hammer during cpu offline
> and doing this repeatedly... say on a 4 socket system, where each socket=16
> logical cpus.
> 
> the system would tend to get hick ups 64 times, since we do the stopmachine
> thread once for each logical cpu. When we want to replace a node for
> reliability reasons, its not clear if this hick ups is a good thing.

Can you provide more detail on what you mean by hiccups?


> Doing kstopmachine() on a single system is in itself noticable, what we heard
> from some OEM's is this would have other app level impact as well.

What "other app level impact"?


> With the bulk removal, we do stop machine just once, but all the 16 cpus
> get removed once hence there is just one hickup, instead of 64.

Have you done any profiling or other instrumentation that identifies
stopmachine as the real culprit here?  I mean, it's a reasonable
assumption to make, but are you sure there's not something else
causing the hiccups?  Perhaps contention on the cpu hotplug lock, or
something wrong in the architecture cpu_disable code?

Module unload also uses stop_machine_run, iirc.  Do you see hiccups
with that, too?


> Less time to offline, avoid process and interrupt bouncing on and off a cpu
> which is just about to be offlined are almost extra fringe benefits you get 
> with the bulk removal approach.

Ok, so that's not the primary motivation for these patches?

Re: [PATCH 0/10] bulk cpu removal support

[Nathan Lynch]

Ashok Raj wrote:
> On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
> Hi Andrew,
> 
> > Shaohua Li <shaohua.li@intel.com> wrote:
> > >
> > > CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
> > 
> > This seems an awful lot of code for something which happens so infrequently.
> > 
> > How big is the problem you're fixing here, and what are the
> > user-observeable effects of these changes?
> 
> This is useful when say a NUMA node is being removed. With new multi-core
> CPUs comming up, considering a 2 core with HT, we could have up to 4 logical
> per socket. On NUMA node with 4 sockets, a node removal will mean we 
> do 16 single cpu offlines. Each time the process and interrupts could
> end up on a CPU that might be removed just immediatly.

But offlining all the cpus in a node is already something that just
works.  If the user is all that concerned about not thrashing the
tasks running on that node, they would have a workload manager that
migrates the tasks off the node before shooting down cpus.  Similar
argument applies to interrupt affinity.

I really haven't seen a compelling argument for why this is needed,
just a bunch of handwaving so far, sorry.

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Thu, May 11, 2006 at 12:19:20PM -0500, Nathan Lynch wrote:
> 
> But offlining all the cpus in a node is already something that just
> works.  If the user is all that concerned about not thrashing the
> tasks running on that node, they would have a workload manager that
> migrates the tasks off the node before shooting down cpus.  Similar
> argument applies to interrupt affinity.
> 
> I really haven't seen a compelling argument for why this is needed,
> just a bunch of handwaving so far, sorry.

Hand waving? Dont think that was intensional though.. i think we are trying
to address a real problem, if there is a reasonable alternate already
that we are not aware of, no problemo...


1. Regarding process migration, someone needs to make sure they run
something like a taskset away from all the cpus that are planned to be
removed upfront. This needs to be done on all processes on the system.

[nick, is there an easier way to do this in today's sched infrastructure or
otherwise]

2. For interrrupt migration, today when we take a cpu offline, we pick 
a random online cpu today. So if you have a cpu going offline, and the 
next logical cpu is also part of the same package, or node, we have
no smarts today to keep migration away from those "to be offlined" cpus.

If we have a solution to these already, or a simpler alternative, we are
open to those... and iam getting help on large system validation.. it might
not be easy right away, but its comming.

Cheers,
ashok


-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [PATCH 0/10] bulk cpu removal support

[Nathan Lynch]

Ashok Raj wrote:
> On Thu, May 11, 2006 at 12:19:20PM -0500, Nathan Lynch wrote:
> > 
> > But offlining all the cpus in a node is already something that just
> > works.  If the user is all that concerned about not thrashing the
> > tasks running on that node, they would have a workload manager that
> > migrates the tasks off the node before shooting down cpus.  Similar
> > argument applies to interrupt affinity.
> > 
> > I really haven't seen a compelling argument for why this is needed,
> > just a bunch of handwaving so far, sorry.
> 
> Hand waving? Dont think that was intensional though.. i think we are trying
> to address a real problem, if there is a reasonable alternate already
> that we are not aware of, no problemo...

If the motivation for these patches is to minimize disruption of the
workload when offlining a group of cpus, then I think the reasonable
alternative is for the admin (or a script) to migrate sensitive
tasks and interrupts to cpus that are not going to be offlined --
before offlining any cpus.

On the other hand, I'm getting the feeling that the problem you're
really trying to address is that offlining lots of cpus takes a long
time (on the order of a couple seconds per cpu) on your architectures.

So is it one of these two things, or a combination of both, or what?


> 1. Regarding process migration, someone needs to make sure they run
> something like a taskset away from all the cpus that are planned to be
> removed upfront. This needs to be done on all processes on the system.

I guess I don't understand what you're getting at here.  Are you
agreeing with me?  Or are you saying that doing this is a problem?


> 2. For interrrupt migration, today when we take a cpu offline, we pick 
> a random online cpu today.

That's an architecture-specific behavior.  On powerpc, interrupts
bound to a dying cpu have their affinity reset to the default (all cpus).


> So if you have a cpu going offline, and the 
> next logical cpu is also part of the same package, or node, we have
> no smarts today to keep migration away from those "to be offlined"
> cpus.

Yes, and the kernel really shouldn't have to be smart about this.
This is a matter of policy.  The admin knows which cpus are going to
be online.  The admin is free to reassign the irqs before offlining
cpus for optimal behavior.  (Of course, the kernel should still have a
sane fallback behavior, and it does.)

So, I still don't see the benefit of adding this change which works
around the kernel's scheduling behavior etc. when the admin easily has
the ability to mitigate the disruption by taking some preliminary
measures.

Re: [PATCH 0/10] bulk cpu removal support

[Ashok Raj]

On Thu, May 11, 2006 at 02:19:39PM -0500, Nathan Lynch wrote:
> > > just a bunch of handwaving so far, sorry.
> > 
> > Hand waving? Dont think that was intensional though.. i think we are trying
> > to address a real problem, if there is a reasonable alternate already
> > that we are not aware of, no problemo...
> 
> If the motivation for these patches is to minimize disruption of the
> workload when offlining a group of cpus, then I think the reasonable
> alternative is for the admin (or a script) to migrate sensitive
> tasks and interrupts to cpus that are not going to be offlined --
> before offlining any cpus.
> 
> On the other hand, I'm getting the feeling that the problem you're
> really trying to address is that offlining lots of cpus takes a long

Think i relied to the other thread... we can invent lots of goodies to keep
process and interrupt bouncing... but nothing is going to stop the 
kstopmachine() hammer for each logical cpu offline in any architecture:-(

RE: [PATCH 0/10] bulk cpu removal support

[Protasevich, Natalie]

Ashok Raj wrote:
> > On Thu, May 11, 2006 at 01:42:47PM -0700, Martin Bligh wrote:
> > > Ashok Raj wrote:
> > > > 
> > > > 
> > > > It depends on whats running at the time... with some 
> light load, i 
> > > > measured wall clock time, i remember seeing 2 secs at 
> times, but 
> > > > its been a long time i did that.. so take that with a pinch :-)_
> > > > 
> > > > i will try to get those idle and load times worked out again... 
> > > > the best i have is a  16 way, if i get help from big 
> system oems i 
> > > > will send the numbers out
> > > 
> > > Why is taking 30s to offline CPUs a problem?
> > > 
> > 
> > Well, the real problem is that for each cpu offline we 
> schedule a RT 
> > thread
> > kstopmachine() on each cpu, then turn off interrupts until this one 
> > cpu has removed. stop_machine_run() is a big enough sledge hammer 
> > during cpu offline and doing this repeatedly... say on a 4 socket 
> > system, where each socket=16 logical cpus.
> > 
> > the system would tend to get hick ups 64 times, since we do the 
> > stopmachine thread once for each logical cpu. When we want 
> to replace 
> > a node for reliability reasons, its not clear if this hick 
> ups is a good thing.
> 
> Can you provide more detail on what you mean by hiccups?
>
> 
> > Doing kstopmachine() on a single system is in itself 
> noticable, what 
> > we heard from some OEM's is this would have other app level 
> impact as well.
> 
Per Ashok's request here is some data on offlining with cpu_bulk_remove
vs. sequentially with a shell script.
I had 64x system (physical CPU) and 128 (those 64 hyperthreaded). The
system was idle. 
Elapsed times are not strikingly different but system/user times are:

                64 CPU                                          128 CPU
(64 physical hyperthreaded)
                cpu_bulk_remove         shell script
cpu_bulk_remove         shell script
all offline     real    0m11.231s       real    0m10.775s       real
0m26.973s       real    0m16.484s
                user    0m0.000s        user    0m0.056         user
0m0.000s        user    0m0.068s
                sys     0m0.072s        sys     0m0.448         sys
0m0.132s        sys     0m1.312s

                real    0m11.977s       real    0m10.550s       real
0m28.978s       real    0m14.259s
                user    0m0.000s        user    0m0.064s        user
0m0.000s        user    0m0.060s
                sys     0m0.032s        sys     0m0.464s        sys
0m0.152s        sys     0m1.432s

32 offline      real    0m1.320s        real    0m2.422s        real
0m1.647s        real    0m1.896s
                user    0m0.000s        user    0m0.000s        user
0m0.000s        user    0m0.020s
                sys     0m0.076s        sys     0m0.232s        sys
0m0.096s        sys     0m0.456s

                real    0m1.407s        real    0m3.348s        real
0m0.418s        real    0m1.198s
                user    0m0.000s        user    0m0.012s        user
0m0.000s        user    0m0.008s
                sys     0m0.072s        sys     0m0.276s        sys
0m0.044s        sys     0m0.244s

groups of 16    real 0m5.877s           real 0m11.403s
                user 0m0.000s           user 0m0.024s
                sys 0m0.140s            sys 0m0.408s

groups of 8     real 0m8.847s           real 0m12.078s          real
0m12.311s       real    0m12.736s
                user 0m0.004s           user 0m0.028s           user
0m0.004s        user    0m0.076s
                sys 0m0.232s            sys 0m0.536s            sys
0m0.448s        sys     0m1.448s

                                                                real
0m11.968s       real    0m14.314s
                                                                user
0m0.008s        user    0m0.084s
                                                                sys
0m0.400s        sys     0m1.492s

With smaller "bulks" cpu_bulk_remove is always better, but with large
ones shell script mostly wins, especially with hyperthreading (despite
of much better system and user times!)

Thanks,
--Natalie

RE: [PATCH 0/10] bulk cpu removal support

[Protasevich, Natalie]

[-- Attachment #1: Type: text/plain, Size: 2929 bytes --]

> Per Ashok's request here is some data on offlining with 
> cpu_bulk_remove vs. sequentially with a shell script.
> I had 64x system (physical CPU) and 128 (those 64 
> hyperthreaded). The system was idle. 
> Elapsed times are not strikingly different but system/user times are:
> 
>                 64 CPU                                        
>   128 CPU
> (64 physical hyperthreaded)
>                 cpu_bulk_remove         shell script
> cpu_bulk_remove         shell script
> all offline     real    0m11.231s       real    0m10.775s       real
> 0m26.973s       real    0m16.484s
>                 user    0m0.000s        user    0m0.056         user
> 0m0.000s        user    0m0.068s
>                 sys     0m0.072s        sys     0m0.448         sys
> 0m0.132s        sys     0m1.312s
> 
>                 real    0m11.977s       real    0m10.550s       real
> 0m28.978s       real    0m14.259s
>                 user    0m0.000s        user    0m0.064s        user
> 0m0.000s        user    0m0.060s
>                 sys     0m0.032s        sys     0m0.464s        sys
> 0m0.152s        sys     0m1.432s
> 
> 32 offline      real    0m1.320s        real    0m2.422s        real
> 0m1.647s        real    0m1.896s
>                 user    0m0.000s        user    0m0.000s        user
> 0m0.000s        user    0m0.020s
>                 sys     0m0.076s        sys     0m0.232s        sys
> 0m0.096s        sys     0m0.456s
> 
>                 real    0m1.407s        real    0m3.348s        real
> 0m0.418s        real    0m1.198s
>                 user    0m0.000s        user    0m0.012s        user
> 0m0.000s        user    0m0.008s
>                 sys     0m0.072s        sys     0m0.276s        sys
> 0m0.044s        sys     0m0.244s
> 
> groups of 16    real 0m5.877s           real 0m11.403s
>                 user 0m0.000s           user 0m0.024s
>                 sys 0m0.140s            sys 0m0.408s
> 
> groups of 8     real 0m8.847s           real 0m12.078s          real
> 0m12.311s       real    0m12.736s
>                 user 0m0.004s           user 0m0.028s           user
> 0m0.004s        user    0m0.076s
>                 sys 0m0.232s            sys 0m0.536s            sys
> 0m0.448s        sys     0m1.448s
> 
>                                                                 real
> 0m11.968s       real    0m14.314s
>                                                                 user
> 0m0.008s        user    0m0.084s
>                                                                 sys
> 0m0.400s        sys     0m1.492s
> 
> With smaller "bulks" cpu_bulk_remove is always better, but 
> with large ones shell script mostly wins, especially with 
> hyperthreading (despite of much better system and user times!)
> 
Oops it all wrapped :O 
I'm attaching the data file instead, sorry about that.

--Natalie
> 

[-- Attachment #2: cpu_data --]
[-- Type: application/octet-stream, Size: 1477 bytes --]

		64 CPU						128 CPU (64 physical hyperthreaded)		
		cpu_bulk_remove		shell script		cpu_bulk_remove		shell script
all offline	real    0m11.231s	real    0m10.775s	real    0m26.973s	real    0m16.484s
		user    0m0.000s	user    0m0.056		user    0m0.000s	user    0m0.068s
		sys     0m0.072s	sys     0m0.448		sys     0m0.132s	sys     0m1.312s
				
		real    0m11.977s	real    0m10.550s	real    0m28.978s	real    0m14.259s
		user    0m0.000s	user    0m0.064s	user    0m0.000s	user    0m0.060s
		sys     0m0.032s	sys     0m0.464s	sys     0m0.152s	sys     0m1.432s
			
32 offline	real    0m1.320s	real    0m2.422s	real    0m1.647s	real    0m1.896s
		user    0m0.000s	user    0m0.000s	user    0m0.000s	user    0m0.020s
		sys     0m0.076s	sys     0m0.232s	sys     0m0.096s	sys     0m0.456s
						
		real    0m1.407s	real    0m3.348s	real    0m0.418s	real    0m1.198s
		user    0m0.000s	user    0m0.012s	user    0m0.000s	user    0m0.008s
		sys     0m0.072s	sys     0m0.276s	sys     0m0.044s	sys     0m0.244s	

groups of 16	real 0m5.877s		real 0m11.403s					
		user 0m0.000s		user 0m0.024s					
		sys 0m0.140s		sys 0m0.408s					
							
groups of 8	real 0m8.847s		real 0m12.078s		real    0m12.311s	real    0m12.736s	
		user 0m0.004s		user 0m0.028s		user    0m0.004s	user    0m0.076s	
		sys 0m0.232s		sys 0m0.536s		sys     0m0.448s	sys     0m1.448s	
						
								real    0m11.968s	real    0m14.314s	
								user    0m0.008s	user    0m0.084s	
								sys     0m0.400s	sys     0m1.492s