Multithreaded /sbin.init? Is it possible?

Multithreaded /sbin.init? Is it possible?

[Eric]

Hello,
	I have an interesting theory. I have beginning programming expereice and 
would like the input of those that have more expereince than me. Please don't 
flame if this is a waste of time. Rather I would like an open discussion on 
the why's and why nots of this.
	Wouldn't the Linux Boot process be speeded up if the init program was 
multithreaded? I know a little about threads.. and I think there might be 
some complication of PPID assignment coming from the threads which would 
probably terminate instead of having a PPID of 1 which is init and will never 
terminate. (i think?). Should init scripts have a parent PID of 1? Does this 
even matter?  
	If I am correct, whenever a programs parent dies, its parent becomes init 
anyways. So when the thread finishes execution of the script, then its parent 
would become init just as if it had started sequentially and directly from 
init.
	The script links in /etc/rc.d/x.d/ would be grouped according to their S and 
K numbers. That is, if you have 5 scripts that are S5, each would be given to 
a thread to execute and there would be 5 threads. The init process would wait 
for each thread to clean up before proceeding to the next group. I know this 
would happen in an undefined order, but if you want, for example, DHCPD to 
start after network then you would assing S5network and S6dhcpd. There would 
still be a defined order for differently numbered scripts, just the bootup 
process wouldnt have to wait for each script to terminate sequentially.
	 Each group would only take as long and the slowest script instead of the sum 
of the execution time of the scripts. The init would start with the S1 
scripts and maybe fork 3 times, wait for each to finish...., then fork maybe 
twice for the S2 group if there are two scripts, wait for them to 
finish...etc....
	Is this even possible?
----------------------
Eric Bambach   
Eric (at) CISU (dot) net 
----------------------

Re: Multithreaded /sbin.init? Is it possible?

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

On Sun, 2003-09-14 12:40:55 -0500, Eric <eric@cisu.net>
wrote in message <200309141240.55800.eric@cisu.net>:
> Hello,
> 	Wouldn't the Linux Boot process be speeded up if the init program was 
> multithreaded? I know a little about threads.. and I think there might be 

Not really if they *run*, but if they sleep...

> 	 Each group would only take as long and the slowest script instead of the sum 
> of the execution time of the scripts. The init would start with the S1 
> scripts and maybe fork 3 times, wait for each to finish...., then fork maybe 
> twice for the S2 group if there are two scripts, wait for them to 
> finish...etc....

Well, if each group would only take to run as long as it's longest
running member, you'd need the CPU power to run them in parallel.
Threads (or multiple, concurrently running /etc/initd./ scripts) would
need several processors to really run in parallel. If you only have one
CPU, all processes/threads would get time slices to run in.

That is, in theory, their behaviour for *running*. However, init scripts
don't "run" all the time. Some of them actually sleep. Sleeping can even
be parallelized in a good manner with less CPUs than processes, so you
may gain a little speed, but from sleeping-in-parallel, not from
running-in-parallel. On the other hand, you hit the system with greater
memory consumption which, for low-RAM machines, may force you into swap,
which will really make you slow...

> 	Is this even possible?

Well, threads (on their own) are a great thing iff

	- you've got many (identical) tasks to _do_ in parallel while
	  having about the same number of CPUs.
	- you need to wait for rare events (ie. read from serial
	  interfaces and the like).

Threads in their own don't speed a well-written program up. They even
have the capacity to slow you down (by requiring unneccessary task
switches).

However, I've done a little test with my laptop (P2-266, 128MB RAM).
Groupwise parallelized, taking rc2 (not rcS) takes 5sec, vs. 20sec for
the non-parallelized traversal. But screen output now is even less
readable than before (Ever seen HP-UX starting?)...

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
      ret = do_actions((curr | FREE_SPEECH) & ~(IRAQ_WAR_2 | DRM | TCPA));

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Multithreaded /sbin.init? Is it possible?

[Eric]

On Sunday 14 September 2003 01:25 pm, Jan-Benedict Glaw wrote:
> On Sun, 2003-09-14 12:40:55 -0500, Eric <eric@cisu.net>
>
> wrote in message <200309141240.55800.eric@cisu.net>:
> > Hello,
> > 	Wouldn't the Linux Boot process be speeded up if the init program was
> > multithreaded? I know a little about threads.. and I think there might be
>
> Not really if they *run*, but if they sleep...
>
True. But I remember seeing my entire boot process halt if a script doesnt 
terminate. Or are you talking about something else. I know that init scripts 
often start other programs in the background. Is this what you mean by sleep?

> > 	 Each group would only take as long and the slowest script instead of
> > the sum of the execution time of the scripts. The init would start with
> > the S1 scripts and maybe fork 3 times, wait for each to finish...., then
> > fork maybe twice for the S2 group if there are two scripts, wait for them
> > to finish...etc....
>
> Well, if each group would only take to run as long as it's longest
> running member, you'd need the CPU power to run them in parallel.
> Threads (or multiple, concurrently running /etc/initd./ scripts) would
> need several processors to really run in parallel. If you only have one
> CPU, all processes/threads would get time slices to run in.
>
> That is, in theory, their behaviour for *running*. However, init scripts
> don't "run" all the time. Some of them actually sleep. Sleeping can even
> be parallelized in a good manner with less CPUs than processes, so you
> may gain a little speed, but from sleeping-in-parallel, not from
> running-in-parallel. On the other hand, you hit the system with greater
> memory consumption which, for low-RAM machines, may force you into swap,
> which will really make you slow...
	I wasn't really concerned about RAM consumption. Im talking about speeding 
things up for a reasonably recent system such as a user desktop box which 
gets rebooted alot instead of a p133 router which never reboots.
>
> > 	Is this even possible?
>
> Well, threads (on their own) are a great thing iff
>
> 	- you've got many (identical) tasks to _do_ in parallel while
> 	  having about the same number of CPUs.
> 	- you need to wait for rare events (ie. read from serial
> 	  interfaces and the like).
>
> Threads in their own don't speed a well-written program up. They even
> have the capacity to slow you down (by requiring unneccessary task
> switches).
>
	True, I do know this much. So far its just a theory that parallelizing will 
speed up boot time on a UniProcessor machine.
	I have a SMP AthlonMP w/ 512MB ram box so running the boot in parallel is 
particularly interesting and useful to me.

> However, I've done a little test with my laptop (P2-266, 128MB RAM).
> Groupwise parallelized, taking rc2 (not rcS) takes 5sec, vs. 20sec for
> the non-parallelized traversal. But screen output now is even less
> readable than before (Ever seen HP-UX starting?)...
>
> MfG, JBG
	Lol, yea I completely understood at the outset that a simple modification of 
init would make the boot process unreadable. However I am taking it in simple 
steps and theorizing about the possible speed benifits, not the actual 
usability yet.  Could you please send me the source or a more descriptive 
e-mail on how you groupwise-parallelized init 2?  Did you modify the init 
source or did you do a little quick hacking and just run the scripts in 
parallel with bash or something?

----------------------
Eric Bambach   
Eric (at) CISU (dot) net 
----------------------

Re: Multithreaded /sbin.init? Is it possible?

[Eric]

> However, I've done a little test with my laptop (P2-266, 128MB RAM).
> Groupwise parallelized, taking rc2 (not rcS) takes 5sec, vs. 20sec for
> the non-parallelized traversal. But screen output now is even less
> readable than before (Ever seen HP-UX starting?)...

	I just had tought after sending my previous e-mail. Would it be possible to 
capture the output of the programs being run? Like each thread would capture 
the output of the program it runs without being printed to the screen and 
then when the thread terminates the main program writes it to the screen. 
That would make the entire message of the program being run atomic and thus 
eliminate odd echo's out of place and done messages conflicting with others. 
Although the messages from each group might be printed in an undefined order, 
you would still get the complete message.
	

----------------------
Eric Bambach   
Eric (at) CISU (dot) net 
----------------------

Re: Multithreaded /sbin.init? Is it possible?

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 5733 bytes --]

On Sun, 2003-09-14 13:59:43 -0500, Eric <eric@cisu.net>
wrote in message <200309141359.43121.eric@cisu.net>:
> On Sunday 14 September 2003 01:25 pm, Jan-Benedict Glaw wrote:
> > On Sun, 2003-09-14 12:40:55 -0500, Eric <eric@cisu.net>
> > wrote in message <200309141240.55800.eric@cisu.net>:
> > > Hello,
> > > 	Wouldn't the Linux Boot process be speeded up if the init program was
> > > multithreaded? I know a little about threads.. and I think there might be
> >
> > Not really if they *run*, but if they sleep...
> >
> True. But I remember seeing my entire boot process halt if a script doesnt 
> terminate. Or are you talking about something else. I know that init scripts 
> often start other programs in the background. Is this what you mean by sleep?

Well, even in your "threatened" (or better named partially parallelized)
run a single script which doesn't finish will halt the whole process
because you're waiting for a *group* to finish...

Wrt. sleep, many scripts work like this:

- Check if binary is there (0.05sec)
- Execute binary (0.1sec)
- sleep 1

For me (Debian/unstable), I find 10 scripts in group "20". Let only half
of them sleep for a second. If you run them one-by-one, you'll have to
sleep for 5 seconds (while all testing for binaries / startup of
binaries take only 0.5sec). If you run all of them in parallel (given
that you've got enough RAM not to be forced into swap), you'll save
4sec...

> > > 	 Each group would only take as long and the slowest script instead of
> > > the sum of the execution time of the scripts. The init would start with
> > > the S1 scripts and maybe fork 3 times, wait for each to finish...., then
> > > fork maybe twice for the S2 group if there are two scripts, wait for them
> > > to finish...etc....
> >
> > Well, if each group would only take to run as long as it's longest
> > running member, you'd need the CPU power to run them in parallel.
> > Threads (or multiple, concurrently running /etc/initd./ scripts) would
> > need several processors to really run in parallel. If you only have one
> > CPU, all processes/threads would get time slices to run in.
> >
> > That is, in theory, their behaviour for *running*. However, init scripts
> > don't "run" all the time. Some of them actually sleep. Sleeping can even
> > be parallelized in a good manner with less CPUs than processes, so you
> > may gain a little speed, but from sleeping-in-parallel, not from
> > running-in-parallel. On the other hand, you hit the system with greater
> > memory consumption which, for low-RAM machines, may force you into swap,
> > which will really make you slow...
> 	I wasn't really concerned about RAM consumption. Im talking about speeding 
> things up for a reasonably recent system such as a user desktop box which 
> gets rebooted alot instead of a p133 router which never reboots.

For these type of boxes, it'll indeed be a speedup to run each numbered
group in parallel.

But this has nothing to do with threads:)

> > > 	Is this even possible?
> >
> > Well, threads (on their own) are a great thing iff
> >
> > 	- you've got many (identical) tasks to _do_ in parallel while
> > 	  having about the same number of CPUs.
> > 	- you need to wait for rare events (ie. read from serial
> > 	  interfaces and the like).
> >
> > Threads in their own don't speed a well-written program up. They even
> > have the capacity to slow you down (by requiring unneccessary task
> > switches).
> >
> 	True, I do know this much. So far its just a theory that parallelizing will 
> speed up boot time on a UniProcessor machine.
> 	I have a SMP AthlonMP w/ 512MB ram box so running the boot in parallel is 
> particularly interesting and useful to me.

ACK. As I told, if you've got enough RAM and a number of groups with > 1
member (ie. Debians group 20 containing 10 members for me), you'll most
probably see a *real* speed-up.

> > However, I've done a little test with my laptop (P2-266, 128MB RAM).
> > Groupwise parallelized, taking rc2 (not rcS) takes 5sec, vs. 20sec for
> > the non-parallelized traversal. But screen output now is even less
> > readable than before (Ever seen HP-UX starting?)...

> 	Lol, yea I completely understood at the outset that a simple modification of 
> init would make the boot process unreadable. However I am taking it in simple 
> steps and theorizing about the possible speed benifits, not the actual 
> usability yet.  Could you please send me the source or a more descriptive 
> e-mail on how you groupwise-parallelized init 2?  Did you modify the init 
> source or did you do a little quick hacking and just run the scripts in 
> parallel with bash or something?

It's quite simple. Debians /etc/init.d/rc script uses a function
"startup". I've modified it so that:

	- The 'Sxx' value is cutted out of the script name and compared
	  to the last value.
	- At command execution, a '&' is added.
	- Looks like this, then:

OLD_S=S00
startup() {
	NEW_S="`echo "$1" | cut -f 4 -d '/' | cut -b 1-3`"
	if [ "${OLD_S}" = "${NEW_S}" ]; then
		wait
		OLD_S="${NEW_S}"
	fi
# ...original startup code, with '&' added to the command execution
}

Additionally, you need a wait after the 'start loop'.

Quite simple, eh? You can do the same to the /etc/init.d/rcS script,
which controls system startup phase (/etc/rcS.d/*).

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
      ret = do_actions((curr | FREE_SPEECH) & ~(IRAQ_WAR_2 | DRM | TCPA));

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Multithreaded /sbin.init? Is it possible?

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Sun, 2003-09-14 13:59:50 -0500, Eric <eric@cisu.net>
wrote in message <200309141359.50596.eric@cisu.net>:
> > However, I've done a little test with my laptop (P2-266, 128MB RAM).
> > Groupwise parallelized, taking rc2 (not rcS) takes 5sec, vs. 20sec for
> > the non-parallelized traversal. But screen output now is even less
> > readable than before (Ever seen HP-UX starting?)...
> 
> 	I just had tought after sending my previous e-mail. Would it be possible to 
> capture the output of the programs being run? Like each thread would capture 
> the output of the program it runs without being printed to the screen and 
> then when the thread terminates the main program writes it to the screen. 
> That would make the entire message of the program being run atomic and thus 
> eliminate odd echo's out of place and done messages conflicting with others. 
> Although the messages from each group might be printed in an undefined order, 
> you would still get the complete message.

That's not much of a problem. You can even get the messages in proper
order...

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
      ret = do_actions((curr | FREE_SPEECH) & ~(IRAQ_WAR_2 | DRM | TCPA));

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Multithreaded /sbin.init? Is it possible?

[szonyi calin]

 --- Eric <eric@cisu.net> a écrit : 
> 	Wouldn't the Linux Boot process be speeded up if the init
> program was 
> multithreaded? 

No
Boot scripts _must_ be executed in a predefined order.
I was playing a liitle with this and i can give you the 
following advice:

1. Bash is the most executed program in the init scripts
to speed up boot process you must speedup bash.
Some people suggest to make statically linked. This is 
my sugestion too. Loading every shared library takes time.

2. Use a script to start your daemons not 20. 
I use Slackware which has one script for each runlevel. 
Believe me. There _is_ a difference compared to Mandrake 
for example. 

3. The disk load has an impact on speed also.
If you start 5 processes in parralel they will start slower
than 5 scripts sequentially because of the loading of every
script in the same time.

4. A bigger processor cache also helps a lot :-)

just my 2 cents.

Sorry for the off-topic answer in linux-c-programming

> ----------------------
> Eric Bambach   
> Eric (at) CISU (dot) net 
> ----------------------

Calin

=====
--
A mouse is a device used to point at 
the xterm you want to type in.
Kim Alm on a.s.r.

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
Yahoo! Mail : http://fr.mail.yahoo.com
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multithreaded /sbin.init? Is it possible?

[Tabris]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 14 September 2003 01:40 pm, Eric wrote:
> Hello,
> 	I have an interesting theory. I have beginning programming expereice
> and would like the input of those that have more expereince than me.
> Please don't flame if this is a waste of time. Rather I would like an
> open discussion on the why's and why nots of this.
> 	Wouldn't the Linux Boot process be speeded up if the init program was
> multithreaded? I know a little about threads.. and I think there might
> be some complication of PPID assignment coming from the threads which
> would probably terminate instead of having a PPID of 1 which is init
> and will never terminate. (i think?). Should init scripts have a parent
> PID of 1? Does this even matter?
> 	If I am correct, whenever a programs parent dies, its parent becomes
> init anyways. So when the thread finishes execution of the script, then
> its parent would become init just as if it had started sequentially and
> directly from init.
> 	The script links in /etc/rc.d/x.d/ would be grouped according to their
> S and K numbers. That is, if you have 5 scripts that are S5, each would
> be given to a thread to execute and there would be 5 threads. The init
> process would wait for each thread to clean up before proceeding to the
> next group. I know this would happen in an undefined order, but if you
> want, for example, DHCPD to start after network then you would assing
> S5network and S6dhcpd. There would still be a defined order for
> differently numbered scripts, just the bootup process wouldnt have to
> wait for each script to terminate sequentially. Each group would only
> take as long and the slowest script instead of the sum of the execution
> time of the scripts. The init would start with the S1 scripts and maybe
> fork 3 times, wait for each to finish...., then fork maybe twice for
> the S2 group if there are two scripts, wait for them to
> finish...etc....
> 	Is this even possible?
> ----------------------
> Eric Bambach
> Eric (at) CISU (dot) net
> ----------------------
Fwiw, there is a project ongoing with this very concept in arklinux, and 
it HAS shown some improvements in boot speed. I don't know exactly how 
much, but the goal is to boot (get to a login screen) in under 30 seconds 
on a desktop type system.

cc:d to the current project maintainer.

- --
tabris
- -- 
Space tells matter how to move and matter tells space how to curve.
		-- Wheeler
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/Ze2YtTgrITXtL+8RAiKdAJ9+3AnkmMdbRiWIwUIP6EYxzAymVACbBAkQ
KWUtU3Gu9+cjcc5jg668pAs=
=gRe5
-----END PGP SIGNATURE-----

[resend] Re: Multithreaded /sbin.init? Is it possible?

[Tabris]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 14 September 2003 01:40 pm, Eric wrote:
> Hello,
> 	I have an interesting theory. I have beginning programming expereice
> and would like the input of those that have more expereince than me.
> Please don't flame if this is a waste of time. Rather I would like an
> open discussion on the why's and why nots of this.
> 	Wouldn't the Linux Boot process be speeded up if the init program was
> multithreaded? I know a little about threads.. and I think there might
> be some complication of PPID assignment coming from the threads which
> would probably terminate instead of having a PPID of 1 which is init
> and will never terminate. (i think?). Should init scripts have a parent
> PID of 1? Does this even matter?
> 	If I am correct, whenever a programs parent dies, its parent becomes
> init anyways. So when the thread finishes execution of the script, then
> its parent would become init just as if it had started sequentially and
> directly from init.
> 	The script links in /etc/rc.d/x.d/ would be grouped according to their
> S and K numbers. That is, if you have 5 scripts that are S5, each would
> be given to a thread to execute and there would be 5 threads. The init
> process would wait for each thread to clean up before proceeding to the
> next group. I know this would happen in an undefined order, but if you
> want, for example, DHCPD to start after network then you would assing
> S5network and S6dhcpd. There would still be a defined order for
> differently numbered scripts, just the bootup process wouldnt have to
> wait for each script to terminate sequentially. Each group would only
> take as long and the slowest script instead of the sum of the execution
> time of the scripts. The init would start with the S1 scripts and maybe
> fork 3 times, wait for each to finish...., then fork maybe twice for
> the S2 group if there are two scripts, wait for them to
> finish...etc....
> 	Is this even possible?
> ----------------------
> Eric Bambach
> Eric (at) CISU (dot) net
> ----------------------
Fwiw, there is a project ongoing with this very concept in arklinux, and
it HAS shown some improvements in boot speed. I don't know exactly how
much, but the goal is to boot (get to a login screen) in under 30 seconds
on a desktop type system.

cc:d to the current project maintainer.

- --
tabris
- --
Space tells matter how to move and matter tells space how to curve.
		-- Wheeler

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/ZgFjtTgrITXtL+8RAjkcAJ44kZWsPjze6CJaw6NeJrbgSX82cQCglZ76
O4wmF1xgOMiBYKYfxtUlOxc=
=oqzs
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Multithreaded /sbin.init? Is it possible?

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 210 bytes --]


Moin!

i have something running which _is_ fast.
not under 30 seconds, but still fast.

it's name is rlmanager and I will upload it to
linux.schottelius.org/rlmanager in the next days.

Have a nico day,

Nico

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]