Linux: the next step

Linux: the next step

[Ariel Faigon]

Hi Linuxies,

So soon the port will be "over".

What's next?

Can we turn this from a neat hacker's adventure into
something really beneficial to SGI?

Thanks to Larry and David, I started collecting some
thoughts in:

	http://info.engr/linux/next-step.html


P.S. I also talked to some people who oppose all this
so I made their voices heard in:

	http://info.engr/linux/skeptic.html

I hope you'll find these two mildly interesting.
I wish TJ could join us in this new endeavor.

Any constructive input is as always welcome.
-- 
Peace, Ariel

Re: Linux: the next step

[Alistair Lambie]

On Aug 1, 12:51pm, Ariel Faigon wrote:
> Subject: Linux: the next step
> Hi Linuxies,
>
> So soon the port will be "over".
>
> What's next?
>
> Can we turn this from a neat hacker's adventure into
> something really beneficial to SGI?
>
> Thanks to Larry and David, I started collecting some
> thoughts in:
>
> 	http://info.engr/linux/next-step.html
>

My 2c...maybe it isn't even worth that much!  I know that there is a lot
of politics here, so some of this may be just dreaming....

I'll break it into the same sections as Ariel has:

We need more contributors to keep momentum:

1. We will HAVE to seed machines to people as I'm guessing most people have
   ~$0 budget for hardware.

2. We need to make sure the basic tools (gcc/glibc) make it as seamless
   as possilble to port things (ie, just configure/make and it happens).
   I think David is doing a great job here, but we are probably going to
   need an ongoing effort from a real good libc hacker who can get any
   compatability/conflict type problems sorted intelligently.  What is
   David's commitment when he leaves...or are we going to let him leave??

3. We need to get some of the neat Indy features supported.  We need
   an SGI graphics wizard to make X fly, then things like ISDN and audio.
   These are the sorts of things that will make people want to use Indy's
   as their primary dev environments.  Fast and great functionality.

4. We need to have some people who are able to commit to help contributors
   who need to find out things.  It needs to be easy for people to find out
   information they need.  This is a tricky area as sometimes people really
   will need to know confidential info...how can we handle this.  I know
   Ralph has found it real difficult at times to find things out...we need
   to make this easier so as not to slow things down.

We should start thinking marketing

1. I believe RedHat would be the best candidate for distribution.  I would
   like to see it 'launched' at some high profile conference (Linux related),
   have it on a CD that has 'awesome' artwork (the old WebForce CD's are a
   good example) and becomes a great collectors item afterwards, and is
   packaged with a real cool T-Shirt for the first X copies.  This stuff
   should be firmly Linux relevant from the design point (penguins??) but
   at the same time be SGI through and through.  SGI should probably
   contribute this part.

2. Don't let marketing name it or we'll get Cosmo Linux....just kidding :-)
   I think we need to emphasis that SGI has put real dollars into this and
   is committed to make it work, rather than the 'faster' bandwagon.  To
   emphasis this point I feel we need to have it run on any new platforms
   at first shipment, and support the neat features.  Nothing will kill it
   quicker than if people get this feeling that this is second tier and
   nobody cares.  We need a number of us to commit to give quality help to
   people on mailing lists/newgroups.  Shucks...maybe if we even had 5% of
   the budget IRIX has...

3. Sponsor Linux events etc....give away Indy's etc...and other things like
   apparel and mugs.  These should all have professional art work that is to
   die for....this stuff works when people really want it.  It should really
   put SGI in front of people.  My feeling is that we can really get in front
   of the technical people in a lot of places and get them on our side.
   While they might not be the decision makers, it sure helps sales people
   if the techo's in the organisation are on side rather than fighting
   against you!

4. A Web/FTP server is a great idea.  Let's get some real great design on the
   Web server though so that the quality is similar to Silicon Surf.  Again,
   it gets the message across that we are serious, committed etc.  We also
   want people accessing as many things with .sgi.com in them!!

5. If we can't get people to come to sgi, how about we sponsor good net
   connections for them that are in a domain linux.sgi.com.  Subtle, but it
   does get sgi associated with them on the net....and we do have a backbone
   to most parts of the world.

>
> P.S. I also talked to some people who oppose all this
> so I made their voices heard in:
>
> 	http://info.engr/linux/skeptic.html
>

One comment I would make here is that people really need to work out whether
we are a hardware or software company.  I realise that this is probably
contentious (sp?), but my feeling is that we are a company who make hardware,
and the software is really a means to an end...selling more hardware.  If this
is true, then people need to look at where we are going to get the greatest
number of hardware sales from....Irix or Linux.  We need both I think to hit
different markets, and maybe we even need NT (don't want to get into that
argument!).  Maybe we should be getting agreement from management for funding
based on sales, and I'll bet we can get a lot more mileage from the funding
than Irix will :-)

> I hope you'll find these two mildly interesting.

It's great having someone pull it all together, thanks.

Cheers, Alistair

-- 
Alistair Lambie					    alambie@wellington.sgi.com
Silicon Graphics New Zealand				  SGI Voicemail: 56791
Level 5, Walsh Wrightson Tower,				    Ph: +64-4-802 1455
94-96 Dixon St, Wellington, NZ			  	   Fax: +64-4-802 1459

Re: Linux: the next step

[Nigel Gamble]

Alistair Lambie wrote:
>    To
>    emphasis this point I feel we need to have it run on any new platforms
>    at first shipment, and support the neat features.

Dream on!  Here in ISD (Indigo2 land), we're working flat out just to
get IRIX to run on any new platforms at first shipment, with all
the neat features supported.  I think you'll find the same
is true of all the other product divisions.

> One comment I would make here is that people really need to work out whether
> we are a hardware or software company.  I realise that this is probably
> contentious (sp?), but my feeling is that we are a company who make hardware,
> and the software is really a means to an end...selling more hardware.  If this
> is true, then people need to look at where we are going to get the greatest
> number of hardware sales from....Irix or Linux.  We need both I think to hit
> different markets, and maybe we even need NT (don't want to get into that
> argument!).  Maybe we should be getting agreement from management for funding
> based on sales, and I'll bet we can get a lot more mileage from the funding
> than Irix will :-)

We are neither a hardware company nor a software company.
We are a *systems* company.  We build the best computer systems
on the planet through a combination of hardware and software
working together.  If we replaced IRIX with NT, we would have
to cancel most of our high performance hardware projects, because
NT will not support the new systems.  As for Linux,
when will it support true symmetric multiprocessing with fine-grained
semaphore and mutex locking, and a fully preemptible kernel?
At the San Diego Usenix earlier this year, Linus Torvalds
thought that the probable answer was "not anytime soon".
(By the way, in order to do this correctly, every device driver would
have to be rewritten.)  Linux would need this *at a minimum* before
there is any hope that it would supplant IRIX on the current
generation of hardware, let alone any future new platforms.

-- 
Nigel Gamble       "Are we going to push the edge of the envelope,
Brain?"
Silicon Graphics   "No, Pinky, but we may get to the sticky part."
nigel@sgi.com
(415) 933-3109

Re: Linux: the next step

[David S. Miller]

   Date: Fri, 02 Aug 1996 16:07:40 -0700
   From: Nigel Gamble <nigel@cthulhu>

   Dream on!  Here in ISD (Indigo2 land), we're working flat out just
   to get IRIX to run on any new platforms at first shipment, with all
   the neat features supported.  I think you'll find the same is true
   of all the other product divisions.

I am one person, I am getting close to bootstrapping a complete Linux
system on the INDY in the course of 3 months.  The only way I could be
dreaming is due to the fact that I have been hacking for 30 hours
straight with no sleep at all and this isn't even flat out for me
actually.

   We are neither a hardware company nor a software company.  We are a
   *systems* company.

Although this may be true, what ever sells your hardware could foot
the purpose correct?  The margin comes from the hardware, and as you
say the software is there to make that hardware capable of doing
something.  If Linux could do this, and sell SGI hardware, SGI is
doing the same thing it is today in my estimation.

   As for Linux, when will it support true symmetric multiprocessing
   with fine-grained semaphore and mutex locking, and a fully
   preemptible kernel?  At the San Diego Usenix earlier this year,
   Linus Torvalds thought that the probable answer was "not anytime
   soon".

Linus, myself, and some other key developers have come to the
conclusion that the current way of doing SMP is no more than an
intellectual curiosity and nothing more.

What he had in the back of his mind when he made that statement at
Usenix was really, "Linux is not going to scale it's SMP in that way."
But being the person that he is, he decided not to put it that way
until he had a cast in stone proof of concept that another way was
indeed possible.

With Linux, thankfully, we have the luxury of really thinking about
how to correctly go about something like a scalable SMP kernel
architecture without a deadline hovering over our heads.

However, I will be frank, and say that no we do not have this SMP of
all SMP figured out yet.  But I do believe that we will come up with
something not short of impressive and cutting edge.

If I had to guess at how we'd go about it, generally?  I would venture
one of the following or a combination thereof perhaps:

1) Scale to 4 cpus and nothing more, cluster past that, run seperate
   kernels on each processor group of 4, which live in their own
   little world and do not have shared resources with the other
   cluster kernels.  With the exception that clusters can pass
   processes between each other when load on one get much too high
   than the others.  (I'm a bit sketchy on any of the real specifics
   on how we would pull this one off entirely)

2) Away with the locks completely, if you are going to do SMP then
   don't add a brick wall to the equation as a means to an end.
   Through numerous conversations with Linus I have deteremined that
   he and I agree that locks all over the place is not the way to go
   and neither is pre-emption.  We believe that the purpose of locks
   can be replaced entirely with carefully designed data structures
   that are atomic via the way they are accessed.

Again these are just off the top of my head, unfortunately I haven't
had the opportunity to get into the zone and put all of my senses into
getting these ideas out of perpetual beta.  I was hoping that if I
got the initial INDY port out of the way and functional very quickly
I'd finally have a chance to think about these things and come up with
something concrete I could actually implement and start playing with.
This is along the lines of what I wanted my big contribution back to
SGI to be for allowing me to have this internship, however it was
obviously necessary to get a proof of concept Linux implementation out
the door on a machine before I could start doing things like that.

My real point is, yes we cannot scale very well on the type of
machines you mention, we do not have fine grained SMP and kernel
preemtion.  The current view I propose is that this method of doing
SMP is one of many oysters that could have been opened to satisfy the
needs of the task at hand.  However I feel that the oyster with
the real pearl of an SMP implementation in it has yet to have been
found.

Putting Linux down because it cannot do what Solaris2.5, SVR4.2MP, and
IRIX can do "today" on very high end hardware I would consider to be
an oxy-moron.  All of it's code has been written by contributors in
their spare time and out of their own good hearts on the low end
hardware they could only get their hands on.  And even with those
constraints, we have things which the other freely available operating
systems simply do not have an probably will not for a very long time.

But now all of this is slowly but surely going to change, as more and
more people like myself get the opportunity to express their hacking
talents on high end hardware (the systems where the capabilities you
mention even matter) via contributions such as my internship and
hardware loans to the various developers, this gap will get smaller
and smaller especially if the current pace of Linux keeps up.  This is
one thing I can assure you of.

Maybe the real issue is, now that I think about, we haven't put our
efforts into crossing that bridge, because we have not gotten to it
yet.  When we do get there, it'll show time baby.

dm@engr.sgi.com

'Ooohh.. "FreeBSD is faster over loopback, when compared to
Linux over the wire". Film at 11.' -Linus

Re: Linux: the next step

[Alistair Lambie]

> 
> Alistair Lambie wrote:
> >    To
> >    emphasis this point I feel we need to have it run on any new platforms
> >    at first shipment, and support the neat features.
> 
> Dream on!  Here in ISD (Indigo2 land), we're working flat out just to
> get IRIX to run on any new platforms at first shipment, with all
> the neat features supported.  I think you'll find the same
> is true of all the other product divisions.
> 
Yeah, I know!!  This may be a dream, but I think it is to some degree achievable.
The thing with IRIX is that EVERYTHING has to be integrated and commercial 
quality on first ship.  What we really need to do in Linux is expose the 
interfaces to things.  If we give enough info to people it will generally 
happen.  I don't know what is involved to do this...maybe a skeleton driver
or something.  It doesn't really matter whether it is optimised, or even bug
free, just as long as it gives those with the ability to write code like this
enough info to work with.  As for applications and things to use this stuff,
thats up to the Linux community to do!

> > One comment I would make here is that people really need to work out whether
> > we are a hardware or software company.  I realise that this is probably
> > contentious (sp?), but my feeling is that we are a company who make hardware,
> > and the software is really a means to an end...selling more hardware.  If this
> > is true, then people need to look at where we are going to get the greatest
> > number of hardware sales from....Irix or Linux.  We need both I think to hit
> > different markets, and maybe we even need NT (don't want to get into that
> > argument!).  Maybe we should be getting agreement from management for funding
> > based on sales, and I'll bet we can get a lot more mileage from the funding
> > than Irix will :-)
> 
> We are neither a hardware company nor a software company.
> We are a *systems* company.  We build the best computer systems
> on the planet through a combination of hardware and software
> working together.  If we replaced IRIX with NT, we would have
> to cancel most of our high performance hardware projects, because
> NT will not support the new systems.  As for Linux,
> when will it support true symmetric multiprocessing with fine-grained
> semaphore and mutex locking, and a fully preemptible kernel?
> At the San Diego Usenix earlier this year, Linus Torvalds
> thought that the probable answer was "not anytime soon".
> (By the way, in order to do this correctly, every device driver would
> have to be rewritten.)  Linux would need this *at a minimum* before
> there is any hope that it would supplant IRIX on the current
> generation of hardware, let alone any future new platforms.
> 
Yup, you're right, we are a systems company.  I don't expect that Linux will 
replace IRIX even in the distant future.  IRIX isn't going to stand still, but
nor is Linux.  One of the things I have noticed about Linux is that when 
someone shows the way, things happen fairly quickly.  Probably Linus's comment
about the symmetric multiprocessing was based on available resource, rather
than a lack of desire to do it.  Maybe this is an opportunity for SGI to 
provide some input into the Linux community...if we were to do some starter
work on this (on one of our boxes of course!!) I'm sure that others would 
pretty soon catch on.  The big thing with Linux is that we don't have to do all
the work....a bit of work on an initial framework and a push in the right 
direction is often all that will be necessary.  Let's take the ISDN port on 
the Indy....if we somehow show people how to use this, I'm sure it won't be
long before someone outside of SGI gets it up and going!  Same with audio.

Let's not think in terms of supplanting IRIX, let's look at growing the number
of boxes we ship.  That's got ot be good for us!!

Cheers, Alistair

Re: Linux: the next step

[Alistair Lambie]

> 
> Alistair Lambie wrote:
> >    To
> >    emphasis this point I feel we need to have it run on any new platforms
> >    at first shipment, and support the neat features.
> 
> Dream on!  Here in ISD (Indigo2 land), we're working flat out just to
> get IRIX to run on any new platforms at first shipment, with all
> the neat features supported.  I think you'll find the same
> is true of all the other product divisions.
> 
Yeah, I know!!  This may be a dream, but I think it is to some degree achievable.
The thing with IRIX is that EVERYTHING has to be integrated and commercial 
quality on first ship.  What we really need to do in Linux is expose the 
interfaces to things.  If we give enough info to people it will generally 
happen.  I don't know what is involved to do this...maybe a skeleton driver
or something.  It doesn't really matter whether it is optimised, or even bug
free, just as long as it gives those with the ability to write code like this
enough info to work with.  As for applications and things to use this stuff,
thats up to the Linux community to do!

> > One comment I would make here is that people really need to work out whether
> > we are a hardware or software company.  I realise that this is probably
> > contentious (sp?), but my feeling is that we are a company who make hardware,
> > and the software is really a means to an end...selling more hardware.  If this
> > is true, then people need to look at where we are going to get the greatest
> > number of hardware sales from....Irix or Linux.  We need both I think to hit
> > different markets, and maybe we even need NT (don't want to get into that
> > argument!).  Maybe we should be getting agreement from management for funding
> > based on sales, and I'll bet we can get a lot more mileage from the funding
> > than Irix will :-)
> 
> We are neither a hardware company nor a software company.
> We are a *systems* company.  We build the best computer systems
> on the planet through a combination of hardware and software
> working together.  If we replaced IRIX with NT, we would have
> to cancel most of our high performance hardware projects, because
> NT will not support the new systems.  As for Linux,
> when will it support true symmetric multiprocessing with fine-grained
> semaphore and mutex locking, and a fully preemptible kernel?
> At the San Diego Usenix earlier this year, Linus Torvalds
> thought that the probable answer was "not anytime soon".
> (By the way, in order to do this correctly, every device driver would
> have to be rewritten.)  Linux would need this *at a minimum* before
> there is any hope that it would supplant IRIX on the current
> generation of hardware, let alone any future new platforms.
> 
Yup, you're right, we are a systems company.  I don't expect that Linux will 
replace IRIX even in the distant future.  IRIX isn't going to stand still, but
nor is Linux.  One of the things I have noticed about Linux is that when 
someone shows the way, things happen fairly quickly.  Probably Linus's comment
about the symmetric multiprocessing was based on available resource, rather
than a lack of desire to do it.  Maybe this is an opportunity for SGI to 
provide some input into the Linux community...if we were to do some starter
work on this (on one of our boxes of course!!) I'm sure that others would 
pretty soon catch on.  The big thing with Linux is that we don't have to do all
the work....a bit of work on an initial framework and a push in the right 
direction is often all that will be necessary.  Let's take the ISDN port on 
the Indy....if we somehow show people how to use this, I'm sure it won't be
long before someone outside of SGI gets it up and going!  Same with audio.

Let's not think in terms of supplanting IRIX, let's look at growing the number
of boxes we ship.  That's got ot be good for us!!

Cheers, Alistair

Re: Linux: the next step

[Larry McVoy]

: Dream on!  Here in ISD (Indigo2 land), we're working flat out just to
: get IRIX to run on any new platforms at first shipment, with all
: the neat features supported.  I think you'll find the same
: is true of all the other product divisions.

It's hard work.  I know people are working hard to keep IRIX going.  And 
there is a lot in IRIX that isn't in Linux.

However, there's a lot in IRIX that shouldn't be in IRIX, let alone Linux.
One of the cool things that David has done this summer was to keep the
generic code free of #ifdef *_WAR stuff that is littered throughout the
IRIX kernel.

: We are neither a hardware company nor a software company.
: We are a *systems* company.  

I agree.  But answer these questions:

	. would anyone buy our hardware if we didn't put the software on it?

	. would anyone buy our software on some other hardware?

I think that gives you an answer to what kind of comapny we really are.

: As for Linux,
: when will it support true symmetric multiprocessing with fine-grained
: semaphore and mutex locking, and a fully preemptible kernel?

The fine grained locking is a lose.  It costs us way too much and what I
am seeing is a trend backwards towards coarser grained locking.  If this
wasn't true, why did we build lego?  Why are we building Nexus?

Linux already has a RT kernel.  I just reviewed a fantastic Usenix paper
that added RT to Linux - fully preemptable, hard real time to < 100 usecs,
and less than 3K lines of code change.  Their test case was a 100Khz clock
pulse on a parallel port; they got it every time, with less than 15usecs
skew, while the system was tar-ing one file system to another, running 
netscape, and generally doing same old Unix stuff just like normal.  It's
really impressive and very uninvasive.

: At the San Diego Usenix earlier this year, Linus Torvalds
: thought that the probable answer was "not anytime soon".
: (By the way, in order to do this correctly, every device driver would
: have to be rewritten.)  

Almost none of the drivers need a rewrite for the RT stuff.  That work was
a good example of what I call "orthogonal thinking" or using a completely
different approach to solve the problem.

: Linux would need this *at a minimum* before
: there is any hope that it would supplant IRIX on the current
: generation of hardware, let alone any future new platforms.

Linux isn't going to replace IRIX any time soon.  However, Linux is
going to show off MTI's chips and SGI's hardware in a somewhat more
positive light.  If all that happens is that people get psyched about
the "hidden" performance in SGI hardware and get IRIX to show off that
performance, then the paltry amount we've spent on Linux will be worth it.

Finally, think about this:  how many times have you had a "great idea"
for better IRIX performance, gone off an prototyped it, only to find
that it makes no difference?  Linux lets you test out those ideas and
see the real performance difference, unadulterated by any surrounding
bloat.  That's cool.  We want that.

Re: Linux: the next step

[Steve Alexander]

lm@gate1-neteng (Larry McVoy) writes:
>The fine grained locking is a lose.  It costs us way too much and what I
>am seeing is a trend backwards towards coarser grained locking.  If this
>wasn't true, why did we build lego?  Why are we building Nexus?

I'd like to see some hard numbers for these assertions.  All of the bottlenecks
in ficus (that I am aware of) have to do with context-switching due to use of
mutexes rather than spinlocks.  The locking overhead itself isn't even on the
map as far as I can tell.

However, I agree that more locking is not the answer.  Eliminating as much
shared, global data as possible is the answer.  Nexus doesn't really do
anything towards that end that I can see.

>Linux already has a RT kernel.  I just reviewed a fantastic Usenix paper
>that added RT to Linux - fully preemptable, hard real time to < 100 usecs,
>and less than 3K lines of code change.  Their test case was a 100Khz clock
>pulse on a parallel port; they got it every time, with less than 15usecs
>skew, while the system was tar-ing one file system to another, running 
>netscape, and generally doing same old Unix stuff just like normal.  It's
>really impressive and very uninvasive.

I'd like to see that when it comes out.

>Finally, think about this:  how many times have you had a "great idea"
>for better IRIX performance, gone off an prototyped it, only to find
>that it makes no difference?  Linux lets you test out those ideas and
>see the real performance difference, unadulterated by any surrounding
>bloat.  That's cool.  We want that.

If it makes no difference, what difference does it make what platform you try
it on?  If it does make a difference, you would be able to measure it despite
the underlying bloat, and again the underlying platform makes no difference.

I'm in favor of some amount of Linux work on SGI gear, but this is a pretty
weak argument.

-- Steve

Re: Linux: the next step

[Steve Alexander]

lm@gate1-neteng (Larry McVoy) writes:
>The fine grained locking is a lose.  It costs us way too much and what I
>am seeing is a trend backwards towards coarser grained locking.  If this
>wasn't true, why did we build lego?  Why are we building Nexus?

I'd like to see some hard numbers for these assertions.  All of the bottlenecks
in ficus (that I am aware of) have to do with context-switching due to use of
mutexes rather than spinlocks.  The locking overhead itself isn't even on the
map as far as I can tell.

However, I agree that more locking is not the answer.  Eliminating as much
shared, global data as possible is the answer.  Nexus doesn't really do
anything towards that end that I can see.

>Linux already has a RT kernel.  I just reviewed a fantastic Usenix paper
>that added RT to Linux - fully preemptable, hard real time to < 100 usecs,
>and less than 3K lines of code change.  Their test case was a 100Khz clock
>pulse on a parallel port; they got it every time, with less than 15usecs
>skew, while the system was tar-ing one file system to another, running 
>netscape, and generally doing same old Unix stuff just like normal.  It's
>really impressive and very uninvasive.

I'd like to see that when it comes out.

>Finally, think about this:  how many times have you had a "great idea"
>for better IRIX performance, gone off an prototyped it, only to find
>that it makes no difference?  Linux lets you test out those ideas and
>see the real performance difference, unadulterated by any surrounding
>bloat.  That's cool.  We want that.

If it makes no difference, what difference does it make what platform you try
it on?  If it does make a difference, you would be able to measure it despite
the underlying bloat, and again the underlying platform makes no difference.

I'm in favor of some amount of Linux work on SGI gear, but this is a pretty
weak argument.

-- Steve

Re: Linux: the next step

[Bob English]

>lm@gate1-neteng (Larry McVoy) writes:
>The fine grained locking is a lose.  It costs us way too much and what I
>am seeing is a trend backwards towards coarser grained locking...

In message <199608122032.NAA17431@refugee.engr.sgi.com> you write:
>I'd like to see some hard numbers for these assertions.  All of the bottlenecks
>in ficus (that I am aware of) have to do with context-switching due to use of
>mutexes rather than spinlocks.  The locking overhead itself isn't even on the
>map as far as I can tell.

If you look at larry's context switch and pipe benchmarks, you'll see
that they run much faster on a single CPU than on two or more.  A lot of
the difference is due to fine-grained locking, which causes more dirty
cache lines to thrash than is actually necessary for the operations
involved.  Pipe communication takes locks at the vnode and inode
level.  Context switches take locks on threads, accounting structures,
and queues.

It all adds up.

>If it makes no difference, what difference does it make what platform you try
>it on?  If it does make a difference, you would be able to measure it despite
>the underlying bloat, and again the underlying platform makes no difference.

Back to the previous discussion.  Getting rid of any one of those locks
wouldn't make much difference.  Getting rid of all of them would, but is
a much taller order.

--bob--

Re: Linux: the next step

[Bob English]

>lm@gate1-neteng (Larry McVoy) writes:
>The fine grained locking is a lose.  It costs us way too much and what I
>am seeing is a trend backwards towards coarser grained locking...

In message <199608122032.NAA17431@refugee.engr.sgi.com> you write:
>I'd like to see some hard numbers for these assertions.  All of the bottlenecks
>in ficus (that I am aware of) have to do with context-switching due to use of
>mutexes rather than spinlocks.  The locking overhead itself isn't even on the
>map as far as I can tell.

If you look at larry's context switch and pipe benchmarks, you'll see
that they run much faster on a single CPU than on two or more.  A lot of
the difference is due to fine-grained locking, which causes more dirty
cache lines to thrash than is actually necessary for the operations
involved.  Pipe communication takes locks at the vnode and inode
level.  Context switches take locks on threads, accounting structures,
and queues.

It all adds up.

>If it makes no difference, what difference does it make what platform you try
>it on?  If it does make a difference, you would be able to measure it despite
>the underlying bloat, and again the underlying platform makes no difference.

Back to the previous discussion.  Getting rid of any one of those locks
wouldn't make much difference.  Getting rid of all of them would, but is
a much taller order.

--bob--

Re: Linux: the next step

[Larry McVoy]

: However, I agree that more locking is not the answer.  Eliminating as much
: shared, global data as possible is the answer.  Nexus doesn't really do
: anything towards that end that I can see.

Yeah.

: >Linux already has a RT kernel.  I just reviewed a fantastic Usenix paper
: >that added RT to Linux - fully preemptable, hard real time to < 100 usecs,
: >and less than 3K lines of code change.  Their test case was a 100Khz clock
: >pulse on a parallel port; they got it every time, with less than 15usecs
: >skew, while the system was tar-ing one file system to another, running 
: >netscape, and generally doing same old Unix stuff just like normal.  It's
: >really impressive and very uninvasive.
: 
: I'd like to see that when it comes out.

You'll have to borrow my copy, Usenix rejected it.  Fuckheads.

: >Finally, think about this:  how many times have you had a "great idea"
: >for better IRIX performance, gone off an prototyped it, only to find
: >that it makes no difference?  Linux lets you test out those ideas and
: >see the real performance difference, unadulterated by any surrounding
: >bloat.  That's cool.  We want that.
: 
: If it makes no difference, what difference does it make what platform you try
: it on?  If it does make a difference, you would be able to measure it despite
: the underlying bloat, and again the underlying platform makes no difference.

Suppose you are trying to reach some performance point, X.  Whatever that
is.  You do some work that would, in theory, get you closer to point X.
It doesn't, you can't see the difference.  Just for the sake of argument,
suppose that that work was indeed correctly focussed on something that
needed to get fixed in order to get to point X.  But the effects of that
work were overshadowed by all of the surrounding bloat.  That doesn't mean
that the work was useless or wrong, it means that the bloat dominates.  If
the bloat wasn't there, you would need to do the work to get to X.

A more concrete example: IRIX syscall entry is something like 15-30 usecs,
if I remember correctly.  On Linux, it's about .5 - 5 usecs.  In the
Linux world, optimizations that made a 100% difference would be lost in
the bloat in IRIX.  Does that mean that those optimizations are pointless?

Not to me, it doesn't.  I want a kernel that is fast enough that dropping
into to diddle something and then popping back out is about a procedure
call, no more.  IRIX is miles from that, Linux is approaching it.

Re: Linux: the next step

[Steve Alexander]

Bob English <renglish@ratliff> writes:
>If you look at larry's context switch and pipe benchmarks, you'll see
>that they run much faster on a single CPU than on two or more.  A lot of
>the difference is due to fine-grained locking, which causes more dirty
>cache lines to thrash than is actually necessary for the operations
>involved.  Pipe communication takes locks at the vnode and inode
>level.  Context switches take locks on threads, accounting structures,
>and queues.

If I look at those benchmarks, they'll probably run faster on Linux on an Indy
than they do on IRIX on an Indy, so I doubt the locking is the big problem we
need to be looking at.  I don't have numbers from David to quote, so maybe I'm
wrong.  I'd like to be wrong.

-- Steve

Re: Linux: the next step

[Steve Alexander]

Bob English <renglish@ratliff> writes:
>If you look at larry's context switch and pipe benchmarks, you'll see
>that they run much faster on a single CPU than on two or more.  A lot of
>the difference is due to fine-grained locking, which causes more dirty
>cache lines to thrash than is actually necessary for the operations
>involved.  Pipe communication takes locks at the vnode and inode
>level.  Context switches take locks on threads, accounting structures,
>and queues.

If I look at those benchmarks, they'll probably run faster on Linux on an Indy
than they do on IRIX on an Indy, so I doubt the locking is the big problem we
need to be looking at.  I don't have numbers from David to quote, so maybe I'm
wrong.  I'd like to be wrong.

-- Steve

Re: Linux: the next step

[Bob English]

In message <199608122118.OAA18053@refugee.engr.sgi.com> you write:
>If I look at those benchmarks, they'll probably run faster on Linux on an Indy
>than they do on IRIX on an Indy, so I doubt the locking is the big problem we
>need to be looking at.  I don't have numbers from David to quote, so maybe I'm
>wrong.  I'd like to be wrong.

I was comparing Irix MP times vs. Irix UP times, and the difference is
both substantial and locking related.  Of course, if Irix were as
efficient on a UP as Linux, the difference might be a factor of 10
rather than a factor of 2 :-).

--bob--

Re: Linux: the next step

[Bob English]

In message <199608122118.OAA18053@refugee.engr.sgi.com> you write:
>If I look at those benchmarks, they'll probably run faster on Linux on an Indy
>than they do on IRIX on an Indy, so I doubt the locking is the big problem we
>need to be looking at.  I don't have numbers from David to quote, so maybe I'm
>wrong.  I'd like to be wrong.

I was comparing Irix MP times vs. Irix UP times, and the difference is
both substantial and locking related.  Of course, if Irix were as
efficient on a UP as Linux, the difference might be a factor of 10
rather than a factor of 2 :-).

--bob--

Re: Linux: the next step

[Steve Alexander]

lm@gate1-neteng (Larry McVoy) writes:
>You'll have to borrow my copy, Usenix rejected it.  Fuckheads.

No doubt to make room for some paper on TCL scripts or something.  Feh.
Yeah, if you can make me a copy or convince the authors to send me one I'd
appreciate it.

>A more concrete example: IRIX syscall entry is something like 15-30 usecs,
>if I remember correctly.  On Linux, it's about .5 - 5 usecs.  In the
>Linux world, optimizations that made a 100% difference would be lost in
>the bloat in IRIX.  Does that mean that those optimizations are pointless?

To some degree, yes.  This is why application developers often have a lot of
platform-specific code; optimizations that work on one system won't necessarily
work on all systems.  I'm not saying this is good, mind you.

-- Steve

Re: Linux: the next step

[Steve Alexander]

lm@gate1-neteng (Larry McVoy) writes:
>You'll have to borrow my copy, Usenix rejected it.  Fuckheads.

No doubt to make room for some paper on TCL scripts or something.  Feh.
Yeah, if you can make me a copy or convince the authors to send me one I'd
appreciate it.

>A more concrete example: IRIX syscall entry is something like 15-30 usecs,
>if I remember correctly.  On Linux, it's about .5 - 5 usecs.  In the
>Linux world, optimizations that made a 100% difference would be lost in
>the bloat in IRIX.  Does that mean that those optimizations are pointless?

To some degree, yes.  This is why application developers often have a lot of
platform-specific code; optimizations that work on one system won't necessarily
work on all systems.  I'm not saying this is good, mind you.

-- Steve

Re: Linux: the next step

[Sandeep Cariapa]

On Aug 12,  2:20pm, Steve Alexander wrote:
> Subject: Re: Linux: the next step
> lm@gate1-neteng (Larry McVoy) writes:
> >You'll have to borrow my copy, Usenix rejected it.  Fuckheads.

No, tell us what you really feel Larry; don't hold back :-)

> No doubt to make room for some paper on TCL scripts or something.  Feh.
> Yeah, if you can make me a copy or convince the authors to send me one I'd
> appreciate it.

Larry, you might want to make this available to the entire Linux mailing
list; I suspect a lot more people would be interested in it. (I am too)
Regards,
Sandeep

>-- End of excerpt from Steve Alexander

Re: Linux: the next step

[John E. Schimmel]

> 
> lm@gate1-neteng (Larry McVoy) writes:
> >You'll have to borrow my copy, Usenix rejected it.  Fuckheads.
> 
> No doubt to make room for some paper on TCL scripts or something.  Feh.
> Yeah, if you can make me a copy or convince the authors to send me one I'd
> appreciate it.

Now wait a minute here.  Read the paper before commenting.
They create a "real-time" thread under Linux.  That does not
mean that it does anything, that the OS has real time support,
etc.  There was nothing particularly new about what it did.
There is now a Linux conference which should showcase this
sort of thing.  Usenix is for new and interesting work that
would be of interest to a large audience.

> 
> -- Steve
> 
> 

------------------------------------------------------------
John E. Schimmel                        jes@sgi.com         
Silicon Graphics Inc.                   Voice: (415)933-4116
http://reality.sgi.com/employees/jes    Fax:   (415)967-8496
------------------------------------------------------------

Re: Linux: the next step

[John E. Schimmel]

> 
> lm@gate1-neteng (Larry McVoy) writes:
> >You'll have to borrow my copy, Usenix rejected it.  Fuckheads.
> 
> No doubt to make room for some paper on TCL scripts or something.  Feh.
> Yeah, if you can make me a copy or convince the authors to send me one I'd
> appreciate it.

Now wait a minute here.  Read the paper before commenting.
They create a "real-time" thread under Linux.  That does not
mean that it does anything, that the OS has real time support,
etc.  There was nothing particularly new about what it did.
There is now a Linux conference which should showcase this
sort of thing.  Usenix is for new and interesting work that
would be of interest to a large audience.

> 
> -- Steve
> 
> 

------------------------------------------------------------
John E. Schimmel                        jes@sgi.com         
Silicon Graphics Inc.                   Voice: (415)933-4116
http://reality.sgi.com/employees/jes    Fax:   (415)967-8496
------------------------------------------------------------

Re: Linux: the next step

[Larry McVoy]

: > No doubt to make room for some paper on TCL scripts or something.  Feh.
: > Yeah, if you can make me a copy or convince the authors to send me one I'd
: > appreciate it.
: 
: Now wait a minute here.  Read the paper before commenting.
: They create a "real-time" thread under Linux.  That does not
: mean that it does anything, that the OS has real time support,
: etc.  There was nothing particularly new about what it did.

Here's what they did:  they took over control of interrupt disable/enable
and dispatch.  They run Linux as a thread and allow you to have 0 or more
real time threads.   If all you are doing is running Linux, it looked
to me like there was minimal impact to the performance of the system.

Real time threads run on the bare hardware, they have little
integration with the OS, other than a pipe like communications device.
The programming paradigm is you run two processes, one real time that
gathers events and stuffs them in the "pipe", and one normal that reads
events out of the pipe and does time _un_critical processing.

They can guarentee real time response of 10Khz, with less than 15 usec
variation.  That was with Linux running a 

	tar f - /usr | (cd /newusr; tar xf -)

a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

I thought the new work was the idea of *not* buggering up your entire OS
to support real time.  They give the real time guys ownership of the CPU
with minimal impact on the rest of the system.  And they had to do next
to nothing to make this work, they just use Unix for everything except
the lowest level of RT work.  I dunno, maybe I'm projecting some pipe
dream, but I think this is really cool work.   Points for orthogonal
thinking on their part.

Re: Linux: the next step

[Larry McVoy]

: > No doubt to make room for some paper on TCL scripts or something.  Feh.
: > Yeah, if you can make me a copy or convince the authors to send me one I'd
: > appreciate it.
: 
: Now wait a minute here.  Read the paper before commenting.
: They create a "real-time" thread under Linux.  That does not
: mean that it does anything, that the OS has real time support,
: etc.  There was nothing particularly new about what it did.

Here's what they did:  they took over control of interrupt disable/enable
and dispatch.  They run Linux as a thread and allow you to have 0 or more
real time threads.   If all you are doing is running Linux, it looked
to me like there was minimal impact to the performance of the system.

Real time threads run on the bare hardware, they have little
integration with the OS, other than a pipe like communications device.
The programming paradigm is you run two processes, one real time that
gathers events and stuffs them in the "pipe", and one normal that reads
events out of the pipe and does time _un_critical processing.

They can guarentee real time response of 10Khz, with less than 15 usec
variation.  That was with Linux running a 

	tar f - /usr | (cd /newusr; tar xf -)

a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

I thought the new work was the idea of *not* buggering up your entire OS
to support real time.  They give the real time guys ownership of the CPU
with minimal impact on the rest of the system.  And they had to do next
to nothing to make this work, they just use Unix for everything except
the lowest level of RT work.  I dunno, maybe I'm projecting some pipe
dream, but I think this is really cool work.   Points for orthogonal
thinking on their part.

Re: Linux: the next step

[Nigel Gamble]

> : Now wait a minute here.  Read the paper before commenting.
> : They create a "real-time" thread under Linux.  That does not
> : mean that it does anything, that the OS has real time support,
> : etc.  There was nothing particularly new about what it did.
> 
> Here's what they did:  they took over control of interrupt disable/enable
> and dispatch.  They run Linux as a thread and allow you to have 0 or more
> real time threads.   If all you are doing is running Linux, it looked
> to me like there was minimal impact to the performance of the system.
> 
> Real time threads run on the bare hardware, they have little
> integration with the OS, other than a pipe like communications device.
> The programming paradigm is you run two processes, one real time that
> gathers events and stuffs them in the "pipe", and one normal that reads
> events out of the pipe and does time _un_critical processing.

That's fine for one class of real-time problems, but it's not going
to help make the RealAudio player (raplayer) useable on Linux.
RealAudio only has to produce AM radio quality audio from a
28.8kbps input stream, yet I only have to move my mouse into
a new window to get audio drop-out.  The RealAudio application,
together with all the support that it is getting from the kernel
in audio drivers, networking code etc., needs to run at a higher
priority than most of the other stuff that might be going on.
This is also a "real time problem", although a different one
from the "get the O/S out of my way" hard real time crowd.

How will Linux solve this type of problem?

> They can guarentee real time response of 10Khz, with less than 15 usec
> variation.  That was with Linux running a 
> 
> 	tar f - /usr | (cd /newusr; tar xf -)
> 
> a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

We can already do this on MP with User Level Interrupts.

As for UP, stay tuned :-)  (Since this mailing list goes outside
of SGI, I'm not going to say any more.)

-- 
Nigel Gamble       "Are we going to push the edge of the envelope, Brain?"
Silicon Graphics   "No, Pinky, but we may get to the sticky part."
nigel@sgi.com
(415) 933-3109

Re: Linux: the next step

[Nigel Gamble]

> : Now wait a minute here.  Read the paper before commenting.
> : They create a "real-time" thread under Linux.  That does not
> : mean that it does anything, that the OS has real time support,
> : etc.  There was nothing particularly new about what it did.
> 
> Here's what they did:  they took over control of interrupt disable/enable
> and dispatch.  They run Linux as a thread and allow you to have 0 or more
> real time threads.   If all you are doing is running Linux, it looked
> to me like there was minimal impact to the performance of the system.
> 
> Real time threads run on the bare hardware, they have little
> integration with the OS, other than a pipe like communications device.
> The programming paradigm is you run two processes, one real time that
> gathers events and stuffs them in the "pipe", and one normal that reads
> events out of the pipe and does time _un_critical processing.

That's fine for one class of real-time problems, but it's not going
to help make the RealAudio player (raplayer) useable on Linux.
RealAudio only has to produce AM radio quality audio from a
28.8kbps input stream, yet I only have to move my mouse into
a new window to get audio drop-out.  The RealAudio application,
together with all the support that it is getting from the kernel
in audio drivers, networking code etc., needs to run at a higher
priority than most of the other stuff that might be going on.
This is also a "real time problem", although a different one
from the "get the O/S out of my way" hard real time crowd.

How will Linux solve this type of problem?

> They can guarentee real time response of 10Khz, with less than 15 usec
> variation.  That was with Linux running a 
> 
> 	tar f - /usr | (cd /newusr; tar xf -)
> 
> a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

We can already do this on MP with User Level Interrupts.

As for UP, stay tuned :-)  (Since this mailing list goes outside
of SGI, I'm not going to say any more.)

-- 
Nigel Gamble       "Are we going to push the edge of the envelope, Brain?"
Silicon Graphics   "No, Pinky, but we may get to the sticky part."
nigel@sgi.com
(415) 933-3109

Re: Linux: the next step

[Bob English]

In message <199608122259.PAA28318@aa5b.engr.sgi.com> you write:
>That's fine for one class of real-time problems, but it's not going
>to help make the RealAudio player (raplayer) useable on Linux...
>The RealAudio application, together with all the support that it is
>getting from the kernel in audio drivers, networking code etc., needs
>to run at a higher priority than most of the other stuff that might be
>going on...
>
>How will Linux solve this type of problem?

With mirrors.  You run another copy of Linux on a high-priority RT
thread.  Full API, fast preemption, minimal performance and code impact.
You work on linux-slow and run rt apps on linux-fast.  You could even
reboot linux-slow without losing your audio signal :-).

--bob--

Re: Linux: the next step

[Bob English]

In message <199608122259.PAA28318@aa5b.engr.sgi.com> you write:
>That's fine for one class of real-time problems, but it's not going
>to help make the RealAudio player (raplayer) useable on Linux...
>The RealAudio application, together with all the support that it is
>getting from the kernel in audio drivers, networking code etc., needs
>to run at a higher priority than most of the other stuff that might be
>going on...
>
>How will Linux solve this type of problem?

With mirrors.  You run another copy of Linux on a high-priority RT
thread.  Full API, fast preemption, minimal performance and code impact.
You work on linux-slow and run rt apps on linux-fast.  You could even
reboot linux-slow without losing your audio signal :-).

--bob--

Re: Linux: the next step

[David S. Miller]

   From: nigel@aa5b (Nigel Gamble)
   Date: Mon, 12 Aug 1996 15:59:04 -0700 (PDT)

   > They can guarentee real time response of 10Khz, with less than 15 usec
   > variation.  That was with Linux running a 
   > 
   > 	tar f - /usr | (cd /newusr; tar xf -)
   > 
   > a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

   We can already do this on MP with User Level Interrupts.

I cannot wait to recompile all of my applications so that they can
take advantage of ULI...

The whole idea behind whatever approach linux will take to anything
(using clone() for threads under Linux is a good example) is that you
will not need to teach all of your "dumb" old programs (even ps!)
about these special processes and or things a process can do.

dm@engr.sgi.com

'Ooohh.. "FreeBSD is faster over loopback, when compared to
Linux over the wire". Film at 11.' -Linus

Re: Linux: the next step

[David S. Miller]

   From: nigel@aa5b (Nigel Gamble)
   Date: Mon, 12 Aug 1996 15:59:04 -0700 (PDT)

   > They can guarentee real time response of 10Khz, with less than 15 usec
   > variation.  That was with Linux running a 
   > 
   > 	tar f - /usr | (cd /newusr; tar xf -)
   > 
   > a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.

   We can already do this on MP with User Level Interrupts.

I cannot wait to recompile all of my applications so that they can
take advantage of ULI...

The whole idea behind whatever approach linux will take to anything
(using clone() for threads under Linux is a good example) is that you
will not need to teach all of your "dumb" old programs (even ps!)
about these special processes and or things a process can do.

dm@engr.sgi.com

'Ooohh.. "FreeBSD is faster over loopback, when compared to
Linux over the wire". Film at 11.' -Linus

Re: Linux: the next step

[Nigel Gamble]

>    From: nigel@aa5b (Nigel Gamble)
>    Date: Mon, 12 Aug 1996 15:59:04 -0700 (PDT)
> 
>    > They can guarentee real time response of 10Khz, with less than 15 usec
>    > variation.  That was with Linux running a 
>    > 
>    > 	tar f - /usr | (cd /newusr; tar xf -)
>    > 
>    > a web browser, X, etc, etc.  Good luck doing that on IRIX up.  Maybe MP.
> 
>    We can already do this on MP with User Level Interrupts.
> 
> I cannot wait to recompile all of my applications so that they can
> take advantage of ULI...

I think you missed my point.  The real time approach that Larry
described will only suit the hard real time programmers who
want only to get the O/S out of the way and program the bare
metal.  The real time application code will have to be custom
written and will get no support from Linux itself - you can't
call Linux system calls from one of these real time threads.
Forget POSIX real time APIs!

IRIX's ULI mechanism already provides a similar facility for
people who are willing to trade O/S support for performance,
(since you can't execute IRIX system calls from a ULI, either).

> The whole idea behind whatever approach linux will take to anything
> (using clone() for threads under Linux is a good example) is that you
> will not need to teach all of your "dumb" old programs (even ps!)
> about these special processes and or things a process can do.

The next release of IRIX will have real time support for user
applications, even on a UP, with guarantees sufficient for
digital media (audio and video).  You won't need to recompile
your application to take advantage of this environment.
We already know what we need to do to achieve this, and have
already implemented most of it.  Our approach is based on a
fully preemptible kernel, interrupts that have a thread context
that can block, and making sure that interrupts are never disabled
for more than 100us.

I'm still waiting to hear "whatever approach linux will take"
to solve this problem.

-- 
Nigel Gamble       "Are we going to push the edge of the envelope, Brain?"
Silicon Graphics   "No, Pinky, but we may get to the sticky part."
nigel@sgi.com
(415) 933-3109

Re: Linux: the next step

[Greg Chesson]

The new real-time facilities in Irix (Nawaf's scheduler) are superior to what
has been available before in any Unix-like system that I know of.  There are
two areas of shortfall, but these are probably unavoidable given the design and
its operating environment:

	1. priority-based scheduling is often a poor substitute for deadline
	   scheduling, although it is useful for many rt applications.

	2. once a process gets scheduled via the rt facility, if it does any
	   system calls you get Irix system call performance.  Sometimes this
	   will be an issue (by limiting the number of syscalls and rt apps),
	   and other times it will not be so important.

I believe that efforts to incorporate hard real-time scheduling into a
full-function OS will satisfy the large majority of customers who have
real-time applications.  These customers do not want to run on the bare iron -
they want their real-time and they want their OS also.

The Linux-based hack that Larry described is indeed elegant - I had a look at
the paper.  However, if that hack were enough to support a wide range of
real-time applications I think system hacks would have implemented it a long
time ago.

g

Re: Linux: the next step

[Greg Chesson]

The new real-time facilities in Irix (Nawaf's scheduler) are superior to what
has been available before in any Unix-like system that I know of.  There are
two areas of shortfall, but these are probably unavoidable given the design and
its operating environment:

	1. priority-based scheduling is often a poor substitute for deadline
	   scheduling, although it is useful for many rt applications.

	2. once a process gets scheduled via the rt facility, if it does any
	   system calls you get Irix system call performance.  Sometimes this
	   will be an issue (by limiting the number of syscalls and rt apps),
	   and other times it will not be so important.

I believe that efforts to incorporate hard real-time scheduling into a
full-function OS will satisfy the large majority of customers who have
real-time applications.  These customers do not want to run on the bare iron -
they want their real-time and they want their OS also.

The Linux-based hack that Larry described is indeed elegant - I had a look at
the paper.  However, if that hack were enough to support a wide range of
real-time applications I think system hacks would have implemented it a long
time ago.

g