Re: what's next for the linux kernel?

Re: what's next for the linux kernel?

[Robert Hancock]

Luke Kenneth Casson Leighton wrote:
> and, what is the linux kernel?
> 
> it's a daft, monolithic design that is suitable and faster on
> single-processor systems, and that design is going to look _really_
> outdated, really soon.

Well, it sounds like it works pretty well on such things as 512 CPU 
Altix systems, so it sounds like the suggestion that Linux is designed 
solely for single-processor systems and isn't suitable for multicore, 
hyperthreaded CPUs doesn't hold much water..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: what's next for the linux kernel?

[Gene Heskett]

On Sunday 02 October 2005 18:43, Robert Hancock wrote:
>Luke Kenneth Casson Leighton wrote:
>> and, what is the linux kernel?
>>
>> it's a daft, monolithic design that is suitable and faster on
>> single-processor systems, and that design is going to look _really_
>> outdated, really soon.
>
>Well, it sounds like it works pretty well on such things as 512 CPU
>Altix systems, so it sounds like the suggestion that Linux is designed
>solely for single-processor systems and isn't suitable for multicore,
>hyperthreaded CPUs doesn't hold much water..

Ahh, yes and no, Robert.  The un-answered question, for that
512 processor Altix system, would be "but does it run things 512
times faster?"  Methinks not, by a very wide margin.  Yes, do a lot
of unrelated things fast maybe, but render a 30 megabyte page with
ghostscript in 10 milliseconds?  Never happen IMO.

And Christoph in the next msg, calls him 1/2 drunk.  He doesn't come
across to me as being more than 1 beer drunk.  And he does make some
interesting points, so if they aren't valid, lets use proveable logic
to shoot them down, not name calling and pointless rhetoric.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: what's next for the linux kernel?

[Vadim Lobanov]

On Sun, 2 Oct 2005, Gene Heskett wrote:

> On Sunday 02 October 2005 18:43, Robert Hancock wrote:
> >Luke Kenneth Casson Leighton wrote:
> >> and, what is the linux kernel?
> >>
> >> it's a daft, monolithic design that is suitable and faster on
> >> single-processor systems, and that design is going to look _really_
> >> outdated, really soon.
> >
> >Well, it sounds like it works pretty well on such things as 512 CPU
> >Altix systems, so it sounds like the suggestion that Linux is designed
> >solely for single-processor systems and isn't suitable for multicore,
> >hyperthreaded CPUs doesn't hold much water..
>
> Ahh, yes and no, Robert.  The un-answered question, for that
> 512 processor Altix system, would be "but does it run things 512
> times faster?"  Methinks not, by a very wide margin.  Yes, do a lot
> of unrelated things fast maybe, but render a 30 megabyte page with
> ghostscript in 10 milliseconds?  Never happen IMO.

This is only true for workloads that are parallelizable. I don't think
any kernel is quite good enough to divine what a single-threaded
userland application is doing and make its work parallel.

That is to say, if we are going to look at examples (and so we should),
then we need to pick an example that is actually expected to benefit
from many-processor machines.

> And Christoph in the next msg, calls him 1/2 drunk.  He doesn't come
> across to me as being more than 1 beer drunk.  And he does make some
> interesting points, so if they aren't valid, lets use proveable logic
> to shoot them down, not name calling and pointless rhetoric.
>
> --
> Cheers, Gene
> "There are four boxes to be used in defense of liberty:
>  soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> 99.35% setiathome rank, not too shabby for a WV hillbilly
> Yahoo.com and AOL/TW attorneys please note, additions to the above
> message by Gene Heskett are:
> Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
>
>
> -

-Vadim Lobanov

Re: what's next for the linux kernel?

[Rik van Riel]

On Sun, 2 Oct 2005, Gene Heskett wrote:

> Ahh, yes and no, Robert.  The un-answered question, for that
> 512 processor Altix system, would be "but does it run things 512
> times faster?"  Methinks not, by a very wide margin.  Yes, do a lot
> of unrelated things fast maybe, but render a 30 megabyte page with
> ghostscript in 10 milliseconds?  Never happen IMO.

You haven't explained us why you think your proposal
would allow Linux to circumvent Amdahl's law...

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Gene Heskett]

On Sunday 02 October 2005 19:48, Rik van Riel wrote:
>On Sun, 2 Oct 2005, Gene Heskett wrote:
>> Ahh, yes and no, Robert.  The un-answered question, for that
>> 512 processor Altix system, would be "but does it run things 512
>> times faster?"  Methinks not, by a very wide margin.  Yes, do a lot
>> of unrelated things fast maybe, but render a 30 megabyte page with
>> ghostscript in 10 milliseconds?  Never happen IMO.
>
>You haven't explained us why you think your proposal
>would allow Linux to circumvent Amdahl's law...

Amdahl's Law?

Thats a reference I don't believe I've been made aware of.  Can you
elaborate?
  
Besides, it isn't my proposal, just a question in that I chose a
scenario (ghostscripts rendering of a page of text) that in fact only
runs maybe 10x faster on an XP-2800 Athlon with a gig of dram than it
did on my old 25 mhz 68040 equipt amiga with 64 megs of dram.  
With 64 megs of dram, so it wasn't nearly as memory bound doing that
as most of the Amiga's were.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: what's next for the linux kernel?

[Jesper Juhl]

On 10/3/05, Gene Heskett <gene.heskett@verizon.net> wrote:
> On Sunday 02 October 2005 19:48, Rik van Riel wrote:
> >On Sun, 2 Oct 2005, Gene Heskett wrote:
> >> Ahh, yes and no, Robert.  The un-answered question, for that
> >> 512 processor Altix system, would be "but does it run things 512
> >> times faster?"  Methinks not, by a very wide margin.  Yes, do a lot
> >> of unrelated things fast maybe, but render a 30 megabyte page with
> >> ghostscript in 10 milliseconds?  Never happen IMO.
> >
> >You haven't explained us why you think your proposal
> >would allow Linux to circumvent Amdahl's law...
>
> Amdahl's Law?
>
http://en.wikipedia.org/wiki/Amdahl's_law
http://home.wlu.edu/~whaleyt/classes/parallel/topics/amdahl.html

And google has even more. Wonderful thing those search engines...


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: what's next for the linux kernel?

[Bodo Eggert]

Marc Perkel <marc@perkel.com> wrote:

[...]
> I'll run through a few ideas here.

> Novell Netware type permissions. ACLs are a step in the right direction
> but Linux isn't any where near where Novell was back in 1990. Linux lets
> you - for example - to delete files that you have no read or write
> access rights to.

It lets you unlink them. That's different from deleting, since the owner
may have his/her private link to that file.

Unlinking is changing the contents of a directory, and it's controlled by
the write permission of the containing directory.

> Netware on the other hand prevents you from deleting
> files that you can't write to and if you have no right it is as if the
> file isn't there.

Imagine a /tmp directory (writable by world) with user "a" creating a file
"foo", umask=077 and user "b" trying to do the same. User "b" will get
'file exists' if he tries to create it, and 'file does not exist' if he
tries to list it. He will go nuts.

BTW: YANI: That about a tmpfs where all-numerical entries can only be
created by the corresponding UID? This would provide a secure, private
tmp directory to each user without the possibility of races and denial-of-
service attacks. Maybe it should be controlled by a mount flag.

> You can't even see it in the directory. Netware also
> has inherited permissions like Windows and Samba has and this is doing
> it right.

You can't do that if you have hardlinks. However, I missed the feature of
overruling file permissions in some special directories, e.g. anything
put under /pub should ignore umask and be a+rX.

> File systems and individual directories should be able to be
> flagged as casesensitive/insensitive.

IMHO not needed.

> Permissions need to be fine
> grained and easy to use. Netware is a good example to emulate.

ACK.

> The bootup sequence of Linux is pathetic. What an ungodly mess.

Which one? The bsd-style, sysv-suse-style, the sysv-debian-style,
the sysv-gentoo-style, the supervise-style, ...?

> The
> FSTAB file needs to go and a smarter system needs to be developed.

Smarter than recognizing the partitions by GUID?

> I
> know this isn't entirely a kernel issue but it is somewhat related.
> 
> I think development needs to be done to make the kernel cleaner and
> smarter rather than just bigger and faster. It's time to look at what
> users need and try to make Linux somewhat more windows like in being
> able to smartly recover from problems.

Using "windows" and "smartly recover from problems" is strange.

> Perhaps better error messages

And it becomes even more strange. Decent error messsages from windows are
as common as snowballs in hell.

> that your traditional kernel panic or hex dump screen of death.

«Some error occured. Press "OK".»

And if there is a help button, it won't.

> The big challenge for Linux is to be able to put it in the hands of
> people who don't want to dedicate their entire life to understanding all
> the little quirks that we have become used to. The slogan should be
> "this just works" and is intuitive.

"Just working" isn't easy if you have zillions of dependencies and even
more possibilities to choose from. You can e.g. make linux use a raid0
of a network block device and a loop-mounted file accessed over a ssh
session as it's root device (just in case if you you got bored of drilling
holes into your knees, pooring milk into them and raising fish in them.)

-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 12:04:01AM +0200, Bodo Eggert wrote:
> > You can't even see it in the directory. Netware also
> > has inherited permissions like Windows and Samba has and this is doing
> > it right.
> 
> You can't do that if you have hardlinks. 

 nt 5.0 added hardlinks to ntfs.

Re: what's next for the linux kernel?

[Nix]

On 5 Oct 2005, Luke Kenneth Casson Leighton murmured woefully:
> On Wed, Oct 05, 2005 at 12:04:01AM +0200, Bodo Eggert wrote:
>> > You can't even see it in the directory. Netware also
>> > has inherited permissions like Windows and Samba has and this is doing
>> > it right.
>> 
>> You can't do that if you have hardlinks. 
> 
>  nt 5.0 added hardlinks to ntfs.

Actually they've been present from the start, but only accessible
through the POSIX subsystem and (IIRC) the Backup API (?!!)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 12:12:49AM +0100, Nix wrote:
> On 5 Oct 2005, Luke Kenneth Casson Leighton murmured woefully:
> > On Wed, Oct 05, 2005 at 12:04:01AM +0200, Bodo Eggert wrote:
> >> > You can't even see it in the directory. Netware also
> >> > has inherited permissions like Windows and Samba has and this is doing
> >> > it right.
> >> 
> >> You can't do that if you have hardlinks. 
> > 
> >  nt 5.0 added hardlinks to ntfs.
> 
> Actually they've been present from the start, but only accessible
> through the POSIX subsystem and (IIRC) the Backup API (?!!)
 
 *hack* *cough* *choke* someone call 911. *thump* too late.

Re: what's next for the linux kernel?

[Nix]

On Thu, 6 Oct 2005, Luke Kenneth Casson Leighton said:
> On Thu, Oct 06, 2005 at 12:12:49AM +0100, Nix wrote:
>> On 5 Oct 2005, Luke Kenneth Casson Leighton murmured woefully:
>> >  nt 5.0 added hardlinks to ntfs.
>> 
>> Actually they've been present from the start, but only accessible
>> through the POSIX subsystem and (IIRC) the Backup API (?!!)
>  
>  *hack* *cough* *choke* someone call 911. *thump* too late.

I agree, it's barmy. (Cygwin, running under the Win32 subsystem as it
does, manipulates hardlinks via the Backup API.)

I assume the `logic' was that even though hardlinks were only there
for the sake of the mostly-useless POSIX subsystem, you still
might want to back them up...

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Nix]

On 4 Oct 2005, Bodo Eggert stated:
> BTW: YANI: That about a tmpfs where all-numerical entries can only be
> created by the corresponding UID? This would provide a secure, private
> tmp directory to each user without the possibility of races and denial-of-
> service attacks. Maybe it should be controlled by a mount flag.

Wouldn't it be less kludgy to just use the existing private namespace
stuff to provide each user with its own /tmp? (Or each user's session,
rather, which is probably much easier, as that corresponds precisely to
one process tree).

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Marc Perkel]

Nix wrote:

>On 4 Oct 2005, Bodo Eggert stated:
>  
>
>>BTW: YANI: That about a tmpfs where all-numerical entries can only be
>>created by the corresponding UID? This would provide a secure, private
>>tmp directory to each user without the possibility of races and denial-of-
>>service attacks. Maybe it should be controlled by a mount flag.
>>    
>>
>
>Wouldn't it be less kludgy to just use the existing private namespace
>stuff to provide each user with its own /tmp? (Or each user's session,
>rather, which is probably much easier, as that corresponds precisely to
>one process tree).
>
>  
>

If you were going to do it right here's what you would do:

People who had files in /tmp would have no rights at all to other users 
/tmp files.
Listing the dirtectory would only display the files you had some access 
to. If you have no rights you don't even see that the file is there.
The effect would be like giving people their own tmp directories.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 07:41:21AM -0700, Marc Perkel wrote:
> If you were going to do it right here's what you would do:
> 
> People who had files in /tmp would have no rights at all to other users 
> /tmp files.
> Listing the dirtectory would only display the files you had some access 
> to. If you have no rights you don't even see that the file is there.
> The effect would be like giving people their own tmp directories.

Except it still wouldn't be able to go: Does file xyz exist?  If not,
create file xyz.  If someone else had xyz that you didn't see, you would
still not be able to create it.  So what is the point of NOT showing it
other than to make it much harder to avoid conflicting names?

if you want to not see files that you have no rights to, filter it in
your user space application when it matters, and let user space see the
files when they need to in order to avoid name conflicts.

It would be an incredibly idiotic system that auto hides files just
because you can't use them.  We have ways to hide files in user space
for the convinience of users.  It would be too inconvinient for
applications if the OS hid files on us.

Len Sorensen

Re: what's next for the linux kernel?

[Marc Perkel]

Lennart Sorensen wrote:

>On Wed, Oct 05, 2005 at 07:41:21AM -0700, Marc Perkel wrote:
>  
>
>>If you were going to do it right here's what you would do:
>>
>>People who had files in /tmp would have no rights at all to other users 
>>/tmp files.
>>Listing the dirtectory would only display the files you had some access 
>>to. If you have no rights you don't even see that the file is there.
>>The effect would be like giving people their own tmp directories.
>>    
>>
>
>Except it still wouldn't be able to go: Does file xyz exist?  If not,
>create file xyz.  If someone else had xyz that you didn't see, you would
>still not be able to create it.  So what is the point of NOT showing it
>other than to make it much harder to avoid conflicting names?
>
>if you want to not see files that you have no rights to, filter it in
>your user space application when it matters, and let user space see the
>files when they need to in order to avoid name conflicts.
>
>It would be an incredibly idiotic system that auto hides files just
>because you can't use them.  We have ways to hide files in user space
>for the convinience of users.  It would be too inconvinient for
>applications if the OS hid files on us.
>
>Len Sorensen
>  
>

What is incredibly idiotic is a file system that allws you to delete 
files that you have no write access to. That is stupid beyond belief and 
only the Unix community doesn't get it.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 07:48:12AM -0700, Marc Perkel wrote:
> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 
> only the Unix community doesn't get it.

If I have a directory and I want to remove it, I can almost always do
that.  The file only goes away if there are no other hardlinks to it.
If someone cares about the file, they should keep a hardlink to it in a
directory THEY own.

Directories within directories on the other hand can make things a pain
since if you don't own the subdir, you can't remove its contents, so you
can't remove it.  You could however likely move the dir somewhere else
to get it out of your way.

My directory is MY file and I get to do whatever I want to it.  Who
knows how someone else managed to get a file into it in the first place.

/tmp is of course different since it has the bit turned on that says
only the file owner can delete it.  If you want that enabled on all
directories, go ahead.  It is supported, although who knows what
applications that might break.  unix supports both ways of directory
behaviour after all.  It isn't one way or the other.

Len Sorensen

Re: what's next for the linux kernel?

[Marc Perkel]

Lennart Sorensen wrote:

>On Wed, Oct 05, 2005 at 07:48:12AM -0700, Marc Perkel wrote:
>  
>
>>What is incredibly idiotic is a file system that allws you to delete 
>>files that you have no write access to. That is stupid beyond belief and 
>>only the Unix community doesn't get it.
>>    
>>
>
>If I have a directory and I want to remove it, I can almost always do
>that.  The file only goes away if there are no other hardlinks to it.
>If someone cares about the file, they should keep a hardlink to it in a
>directory THEY own.
>
>Directories within directories on the other hand can make things a pain
>since if you don't own the subdir, you can't remove its contents, so you
>can't remove it.  You could however likely move the dir somewhere else
>to get it out of your way.
>
>My directory is MY file and I get to do whatever I want to it.  Who
>knows how someone else managed to get a file into it in the first place.
>
>/tmp is of course different since it has the bit turned on that says
>only the file owner can delete it.  If you want that enabled on all
>directories, go ahead.  It is supported, although who knows what
>applications that might break.  unix supports both ways of directory
>behaviour after all.  It isn't one way or the other.
>
>Len Sorensen
>  
>

Agian - thinking outside the box.

If the permissions were don'e right in your own directories your 
inherited rights would give your permissions automatically to your home 
directory and all directories uner it. Netware has a concept called an 
inherited rights mask - something Linux lacks. Windows also has rights 
like this and Samba emulates it. So unless root put files in your 
directory and specifically denied you rights to them, you would have 
full rights to your own directory.

However - if you were browsing the /etc directory and there were files 
there that you had no read or write access to - then you wouldn't even 
be able to list them. If you went to the home directory and lets say 
everyone had 700 permissions on all the directories withing home, you 
would only see your own directory. You wouldn't even be able to know 
what other directories existed there.

If you want to start thinking about DOING IT RIGHT you need to think 
beyond the Unix model and start looking at Netware. Maybe in 5 years 
Linux will evolve to where Netware was in 1990.

Unix permissions totally suck but it's old baggage that you're stuck 
with somewhat. Are you going to be stuck forever and is Linux ever going 
to grow up and move on to better things? Linux is crippled when it comes 
to permissions. The Windows people are laughing at you and you don't 
even get it why they are laughing.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 08:08:26AM -0700, Marc Perkel wrote:
> Agian - thinking outside the box.
> 
> If the permissions were don'e right in your own directories your 
> inherited rights would give your permissions automatically to your home 
> directory and all directories uner it. Netware has a concept called an 
> inherited rights mask - something Linux lacks. Windows also has rights 
> like this and Samba emulates it. So unless root put files in your 
> directory and specifically denied you rights to them, you would have 
> full rights to your own directory.

Well I could have setup my home dirs with ACL to have inherited rights
to all files created under it for the owner.  Well I didn't but then
again I don't allow other people to write to my home dir so it hasn't
been a problem.

> However - if you were browsing the /etc directory and there were files 
> there that you had no read or write access to - then you wouldn't even 
> be able to list them. If you went to the home directory and lets say 
> everyone had 700 permissions on all the directories withing home, you 
> would only see your own directory. You wouldn't even be able to know 
> what other directories existed there.

Well I don't have write access to /etc so who cares.  I do have write
access to /tmp and there is matters that I can list files I have no
access to.  Hence why /tmp IS readable by all.  If I couldn't list files
there I would have to randomly try filenames until I found one that
wasn't already in use but invisible to me due to idiot magic.

What error does netware return if you try to create a file in a
directory that already exists but which you can't see?  Any error would
be indirectly a way to see the file, so it should have just been visible
in the first place.

> If you want to start thinking about DOING IT RIGHT you need to think 
> beyond the Unix model and start looking at Netware. Maybe in 5 years 
> Linux will evolve to where Netware was in 1990.

NetWare was not the be all and end all of filesystems.  Far from it.  It
had some good points, but it certainly didn't solve everything.

> Unix permissions totally suck but it's old baggage that you're stuck 
> with somewhat. Are you going to be stuck forever and is Linux ever going 
> to grow up and move on to better things? Linux is crippled when it comes 
> to permissions. The Windows people are laughing at you and you don't 
> even get it why they are laughing.

I find unix permissions work great in general, and for complex things
posix ACL has done everything I wanted it to and works great with samba.

The behaviours you claim are so great in netware to me seem like things
I would very much NOT want to have to deal with.  I like things simple
enough to understand them so I can make sure they are right.

Len Sorensen

Re: what's next for the linux kernel?

[Nix]

On Wed, 05 Oct 2005, Marc Perkel yowled:
> Agian - thinking outside the box.

I hate that phrase. There is no `box'.

> If the permissions were don'e right in your own directories your
> inherited rights would give your permissions automatically to your
> home directory and all directories uner it. Netware has a concept
> called an inherited rights mask - something Linux lacks. Windows also
> has rights like this and Samba emulates it. So unless root put files
> in your directory and specifically denied you rights to them, you
> would have full rights to your own directory.

So, um, what happens to these permissions when you copy a file and put
it somewhere else? Do the inherited rights go with it or not? In Unix
it's pretty intuitive. In this system there seem to be two right
answers, both of which seem... risky from a security perspective.

> However - if you were browsing the /etc directory and there were files
> there that you had no read or write access to - then you wouldn't even
> be able to list them.

/tmp is the problem here, and shows the futility and pointlessness of
this feature. If you have an unlistable file in /tmp, *its name is still
determinable*, because other users cannot create files with that
name. The concept adds *nothing* over some combination of dirs with the
execute bit cleared for some set of users and subdirectories which
cannot be read by some set of users. There's no need for this profoundly
non-Unixlike permission at all. (As usual, ACLs make managing this on
a fine-grained scale rather easier.)

>                       If you went to the home directory and lets say
> everyone had 700 permissions on all the directories withing home, you
> would only see your own directory. You wouldn't even be able to know
> what other directories existed there.

This is what per-process filesystems are for.

> If you want to start thinking about DOING IT RIGHT you need to think
> beyond the Unix model and start looking at Netware. Maybe in 5 years
> Linux will evolve to where Netware was in 1990.

I think Plan 9 is a better goal than Netware. At least it was designed
by people aiming for a better Unix rather than people trying to build a
better DOS, and so is more likely to have a compatible philosophy.

> Unix permissions totally suck but it's old baggage that you're stuck
> with somewhat. Are you going to be stuck forever and is Linux ever
> going to grow up and move on to better things? Linux is crippled when
> it comes to permissions.

Well, you can't change it drastically without violating POSIX. There's
no damned way Linux is going to do *that*.

>                          The Windows people are laughing at you and
> you don't even get it why they are laughing.

You *do* realise just how incapable the Windows permission-management
GUI is, don't you? Any OS where the command-line tools hide half
the permissions model and the GUI hides a slightly different half,
and where looking at a set of permissions and hitting cancel can
*change* those permissions drastically, is not sane.

(Disclaimer: the last time I bothered to verify the latter behaviour
was in NT4. Maybe they've partially fixed it.)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Marc Perkel]

Nix wrote:

>On Wed, 05 Oct 2005, Marc Perkel yowled:
>  
>
>>Agian - thinking outside the box.
>>    
>>
>
>I hate that phrase. There is no `box'.
>
>  
>
That's what it looks like when you are inside it.

>>If the permissions were don'e right in your own directories your
>>inherited rights would give your permissions automatically to your
>>home directory and all directories uner it. Netware has a concept
>>called an inherited rights mask - something Linux lacks. Windows also
>>has rights like this and Samba emulates it. So unless root put files
>>in your directory and specifically denied you rights to them, you
>>would have full rights to your own directory.
>>    
>>
>
>So, um, what happens to these permissions when you copy a file and put
>it somewhere else? Do the inherited rights go with it or not? In Unix
>it's pretty intuitive. In this system there seem to be two right
>answers, both of which seem... risky from a security perspective.
>  
>
You inherit the rights of the new directory.

Also - under Netware not all permissions are stored with the file. The 
rights are calculated from the file heirachy so you don't store a lot of 
data with each file unless the file has permissions set that is 
different than that of the directory it's in. So moving a file to 
someone's home directory doesn't require any permissions to be set to 
give the user rights to the file.

>  
>
>>However - if you were browsing the /etc directory and there were files
>>there that you had no read or write access to - then you wouldn't even
>>be able to list them.
>>    
>>
>
>/tmp is the problem here, and shows the futility and pointlessness of
>this feature. If you have an unlistable file in /tmp, *its name is still
>determinable*, because other users cannot create files with that
>name. The concept adds *nothing* over some combination of dirs with the
>execute bit cleared for some set of users and subdirectories which
>cannot be read by some set of users. There's no need for this profoundly
>non-Unixlike permission at all. (As usual, ACLs make managing this on
>a fine-grained scale rather easier.)
>
>  
>
It doesn't really make sense to use the /tmp directory the way Unix uses 
it. Why would you want just anyone to even know the names of the 
temporary files you are using. Users should have their own temp 
directory or create their own directory within /tmp

But - to address your question - if there were an invisible (to you) 
file in a directory that you had create rights to then you would get a 
file creation error.

>>                      If you went to the home directory and lets say
>>everyone had 700 permissions on all the directories withing home, you
>>would only see your own directory. You wouldn't even be able to know
>>what other directories existed there.
>>    
>>
>
>This is what per-process filesystems are for.
>
>  
>
>>If you want to start thinking about DOING IT RIGHT you need to think
>>beyond the Unix model and start looking at Netware. Maybe in 5 years
>>Linux will evolve to where Netware was in 1990.
>>    
>>
>
>I think Plan 9 is a better goal than Netware. At least it was designed
>by people aiming for a better Unix rather than people trying to build a
>better DOS, and so is more likely to have a compatible philosophy.
>
>  
>
I'm not familiar with Plan 9.

>>Unix permissions totally suck but it's old baggage that you're stuck
>>with somewhat. Are you going to be stuck forever and is Linux ever
>>going to grow up and move on to better things? Linux is crippled when
>>it comes to permissions.
>>    
>>
>
>Well, you can't change it drastically without violating POSIX. There's
>no damned way Linux is going to do *that*.
>
>  
>
>>                         The Windows people are laughing at you and
>>you don't even get it why they are laughing.
>>    
>>
>
>You *do* realise just how incapable the Windows permission-management
>GUI is, don't you? Any OS where the command-line tools hide half
>the permissions model and the GUI hides a slightly different half,
>and where looking at a set of permissions and hitting cancel can
>*change* those permissions drastically, is not sane.
>  
>
That's why I'm pushing netware as a model rather than windows. But 
Windows file permissions are superior to Linux.

>(Disclaimer: the last time I bothered to verify the latter behaviour
>was in NT4. Maybe they've partially fixed it.)
>
>  
>

One place where Windows wins over Linux is in the "easy to use" 
category. Something the Linux world should look ast.

I am a Linux supporter and love it. I'm saying this to help make it better.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Nix]

On Wed, 05 Oct 2005, Marc Perkel stipulated:
> Nix wrote:
>>On Wed, 05 Oct 2005, Marc Perkel yowled:
>>So, um, what happens to these permissions when you copy a file and put
>>it somewhere else? Do the inherited rights go with it or not? In Unix
>>it's pretty intuitive. In this system there seem to be two right
>>answers, both of which seem... risky from a security perspective.
>
> You inherit the rights of the new directory.

That's a hugely counterintuitive potential security hole in waiting.

Because any number of permissions can be inherited (anything from none
of them, to all of them), this means that upon copying a file *you
cannot reliably tell what its permissions will become* without checking
it.

I don't think I need to point out the flaw in such a scheme.

> Also - under Netware not all permissions are stored with the file. The
> rights are calculated from the file heirachy so you don't store a lot
> of data with each file unless the file has permissions set that is
> different than that of the directory it's in. So moving a file to
> someone's home directory doesn't require any permissions to be set to
> give the user rights to the file.

I consider this a substantial *disadvantage*. Changing permissions should
be something that you have to explicitly do: it shouldn't happen quietly
behind your back merely on account of a rename().

>>/tmp is the problem here, and shows the futility and pointlessness of
>>this feature. If you have an unlistable file in /tmp, *its name is still
>>determinable*, because other users cannot create files with that
>>name. The concept adds *nothing* over some combination of dirs with the
>>execute bit cleared for some set of users and subdirectories which
>>cannot be read by some set of users. There's no need for this profoundly
>>non-Unixlike permission at all. (As usual, ACLs make managing this on
>>a fine-grained scale rather easier.)
>
> It doesn't really make sense to use the /tmp directory the way Unix
> uses it. Why would you want just anyone to even know the names of the
> temporary files you are using. Users should have their own temp
> directory or create their own directory within /tmp

This is what per-process namespaces are for.

> But - to address your question - if there were an invisible (to you)
> file in a directory that you had create rights to then you would get a
> file creation error.

i.e., the filename is *not* invisible, but trivially determinable by a
sufficiently determined attacker; sticking the file in a non-readable
directory is doable *now*, and *already* lacks this weakness.

i.e., the feature is useless.

>>I think Plan 9 is a better goal than Netware. At least it was designed
>>by people aiming for a better Unix rather than people trying to build a
>>better DOS, and so is more likely to have a compatible philosophy.
>>
> I'm not familiar with Plan 9.

Linux seems to be stealing all its good ideas bit by bit, so you'll
know sooner or later. :)

But per-process namespaces are *wonderful*. Goodbye, PATH, for instance:
just mount every binary this user should see in this user's /bin. He
gets his own /tmp, and /home contains *his home directory*.

(Special considerations have to be made for privileged processes: we
don't want the user to be able to fake them out by fooling with /etc,
say, or with their shared libraries. Perhaps setuid programs revert to a
standard namespace, and can see the user's one in
/proc/self/nonprivileged-root or something.)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Denis Vlasenko]

On Wednesday 05 October 2005 22:30, Marc Perkel wrote:
> >So, um, what happens to these permissions when you copy a file and put
> >it somewhere else? Do the inherited rights go with it or not? In Unix
> >it's pretty intuitive. In this system there seem to be two right
> >answers, both of which seem... risky from a security perspective.
> >  
> >
> You inherit the rights of the new directory.
> 
> Also - under Netware not all permissions are stored with the file. The 
> rights are calculated from the file heirachy so you don't store a lot of 
> data with each file unless the file has permissions set that is 

Since when 16 bits is a lot of data?

> different than that of the directory it's in. So moving a file to 
> someone's home directory doesn't require any permissions to be set to 
> give the user rights to the file.
> 
> >/tmp is the problem here, and shows the futility and pointlessness of
> >this feature. If you have an unlistable file in /tmp, *its name is still
> >determinable*, because other users cannot create files with that
> >name. The concept adds *nothing* over some combination of dirs with the
> >execute bit cleared for some set of users and subdirectories which
> >cannot be read by some set of users. There's no need for this profoundly
> >non-Unixlike permission at all. (As usual, ACLs make managing this on
> >a fine-grained scale rather easier.)
> >
> It doesn't really make sense to use the /tmp directory the way Unix uses 
> it. Why would you want just anyone to even know the names of the 
> temporary files you are using. Users should have their own temp 
> directory or create their own directory within /tmp

Then use ~/tmp in your homedir. What's the problem?

> >You *do* realise just how incapable the Windows permission-management
> >GUI is, don't you? Any OS where the command-line tools hide half
> >the permissions model and the GUI hides a slightly different half,
> >and where looking at a set of permissions and hitting cancel can
> >*change* those permissions drastically, is not sane.
> >  
> >
> That's why I'm pushing netware as a model rather than windows. But 
> Windows file permissions are superior to Linux.
> 
> >(Disclaimer: the last time I bothered to verify the latter behaviour
> >was in NT4. Maybe they've partially fixed it.)
> 
> One place where Windows wins over Linux is in the "easy to use" 
> category. Something the Linux world should look ast.

"Easy to use" to whom?

Command line tools for changing permissions under Windows are anything
but "easy to use". They are also feature incomplete. Tools for changing
ownership do not exist at all. Tools for changing permissions/ownership
on registry do not exist also. Maybe third party ones exist, dunno.
Every one that I saw on the Inet, was either buggy, incomplete, or both.

For admins, this is a huge hole in usability.

> I am a Linux supporter and love it. I'm saying this to help make it better.
--
vda

Re: what's next for the linux kernel?

[Al Viro]

On Wed, Oct 05, 2005 at 08:16:19PM +0100, Nix wrote:
> On Wed, 05 Oct 2005, Marc Perkel yowled:
> > If you want to start thinking about DOING IT RIGHT you need to think
> > beyond the Unix model and start looking at Netware. Maybe in 5 years
> > Linux will evolve to where Netware was in 1990.

Ugly as hell and thankfully about to go extinct outside of dark places
where its malignant presense still lingers amidst the Shoggoth droppings?

> I think Plan 9 is a better goal than Netware. At least it was designed
> by people aiming for a better Unix rather than people trying to build a
> better DOS, and so is more likely to have a compatible philosophy.

s/aiming.*/with taste that doesn't belong in R'Lyeh/

Re: what's next for the linux kernel?

[Marc Perkel]

Al Viro wrote:

>On Wed, Oct 05, 2005 at 08:16:19PM +0100, Nix wrote:
>  
>
>>On Wed, 05 Oct 2005, Marc Perkel yowled:
>>    
>>
>>>If you want to start thinking about DOING IT RIGHT you need to think
>>>beyond the Unix model and start looking at Netware. Maybe in 5 years
>>>Linux will evolve to where Netware was in 1990.
>>>      
>>>
>
>Ugly as hell and thankfully about to go extinct outside of dark places
>where its malignant presense still lingers amidst the Shoggoth droppings?
>
>  
>
The big boys are still running Netware because Linux doesn't do the 
things they need netware to do. And Netware isn't going away until Linux 
has the power to replace it. Linux permissions acient and awkward. 
Netware is wonderful. Linux is crippled. Linux needs to modernize.

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 12:49:12PM -0700, Marc Perkel wrote:
> The big boys are still running Netware because Linux doesn't do the 
> things they need netware to do. And Netware isn't going away until Linux 
> has the power to replace it. Linux permissions acient and awkward. 
> Netware is wonderful. Linux is crippled. Linux needs to modernize.

Netware is a fileserver for windows/dos/os2, linux is an operating
system for running applications.  Being able to serve files to other
systems is just one potential application to run on linux but it has
plenty of other things it can do.

People that need netware are people that need a fileserver for their
windows systems.  Linux is probably not what they need then.

Len Sorensen

Re: what's next for the linux kernel?

[linux-os (Dick Johnson)]

On Wed, 5 Oct 2005, Al Viro wrote:

> On Wed, Oct 05, 2005 at 08:16:19PM +0100, Nix wrote:
>> On Wed, 05 Oct 2005, Marc Perkel yowled:
>>> If you want to start thinking about DOING IT RIGHT you need to think
>>> beyond the Unix model and start looking at Netware. Maybe in 5 years
>>> Linux will evolve to where Netware was in 1990.
>
> Ugly as hell and thankfully about to go extinct outside of dark places
> where its malignant presense still lingers amidst the Shoggoth droppings?
>
>> I think Plan 9 is a better goal than Netware. At least it was designed
>> by people aiming for a better Unix rather than people trying to build a
>> better DOS, and so is more likely to have a compatible philosophy.
>
> s/aiming.*/with taste that doesn't belong in R'Lyeh/
>

Excuse me but wasn't Netware the DOS server that started as
a corrupted BSD Unix system, purposely obfuscated to prevent
prying eyes from discovering that it was an unauthorized Unix
variant?

Or was it really a variant of the defunct Xerox XNS that claimed
that they created SPX/IPX?


Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: what's next for the linux kernel?

[Nix]

On Wed, 05 Oct 2005, Marc Perkel announced authoritatively:
> If the permissions were don'e right in your own directories your
> inherited rights would give your permissions automatically to your
> home directory and all directories uner it. Netware has a concept
> called an inherited rights mask - something Linux lacks. Windows also
> has rights like this and Samba emulates it. So unless root put files
> in your directory and specifically denied you rights to them, you
> would have full rights to your own directory.

I quite forgot the most grotesque problem with this, which in my opinion
completely eliminates the possibility of inherited permissions on any
even vaguely-POSIX-like system.

What happens if you have a file with two links in directories with
different inheritable permissions set? What are its permissions now?
Surely it doesn't depend on where you happened to open it from!

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Nigel Rantor]

Marc Perkel wrote:
> 
> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 
> only the Unix community doesn't get it.

No.

What's idiotic is leaving your property (files) on someone elses desk 
(directory) and expecting them always to be there when you come back.

   n

Re: what's next for the linux kernel?

[Bodo Eggert]

On Wed, 5 Oct 2005, Marc Perkel wrote:

> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 
> only the Unix community doesn't get it.

1) Unlinking is not deleting, it may just trigger deleting. !Unix people 
   don't get it.

   If you like non-owners to have no unlink permission, you have to set 
   the sticky bit. If you want other users not to be able to delete files
   you put into a public directory, put a link into that directory and 
   keep one in a private directory.

2) If you're accounted for the space occupied by your directories, you 
   need the permission to remove the directory. Otherwise you could DoS 
   other users if you have write permission in one of his directories.

-- 
Funny quotes:
29. When someone asks you, "A penny for your thoughts" and you put your two
    cents in . . . what happens to the other penny?

Re: what's next for the linux kernel?

[Florin Malita]

On Wed, 05 Oct 2005 07:48:12 -0700
Marc Perkel <marc@perkel.com> wrote:
> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 
> only the Unix community doesn't get it.

It stops being idiotic as soon as you realize that _deleting_ a
file doesn't involve _writing_ to it in any way. It's not about UNIX,
it's about common sense - try thinking outside of the Netware box for a
sec ;)

Florin

Re: what's next for the linux kernel?

[Marc Perkel]

Florin Malita wrote:

>On Wed, 05 Oct 2005 07:48:12 -0700
>Marc Perkel <marc@perkel.com> wrote:
>  
>
>>What is incredibly idiotic is a file system that allws you to delete 
>>files that you have no write access to. That is stupid beyond belief and 
>>only the Unix community doesn't get it.
>>    
>>
>
>It stops being idiotic as soon as you realize that _deleting_ a
>file doesn't involve _writing_ to it in any way. It's not about UNIX,
>it's about common sense - try thinking outside of the Netware box for a
>sec ;)
>
>Florin
>  
>
What you don't get is that if you don't have rights to write to a file 
then you shouldn't have the right to delete the file.  Once you get past 
the "inside the box" Unix thinking you'll see the logic in this. So what 
if the process of deleting a file involves writing to it. That's not 
relevant.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 12:44:25PM -0700, Marc Perkel wrote:
> What you don't get is that if you don't have rights to write to a file 
> then you shouldn't have the right to delete the file.  Once you get past 
> the "inside the box" Unix thinking you'll see the logic in this. So what 
> if the process of deleting a file involves writing to it. That's not 
> relevant.

When a system supports hardlinks, it IS relevant.

So if I decide I want a link to a file like say /etc/group in my home
dir (let us pretend they are on the same partition) so I make a hardlink
to it in my home dir and end up with a file still owned by root (since I
shouldn't be able to add me as owner to the file just by linking to it
after all).  Should I now have to go bother the admin about deleting the
file from my home dir if I decide that wasn't really what I wanted?  If
I didn't have write permissions to the dir I wouldn't have been able to
make the link in the first place, so since I made it I should be able to
delete it, and I can with the unix way of doing things.  I still can't
edit it anymore than I could in the original place since it linked with
the new link to the file having the excact same permissions as the
original.  Only someone like root can go chance the owner of a hardlink
to someone else and start setting up some interesting file permissions
using multiple hardlinks to one file.

I suspect you can't do that on netware, so you would have to add
explicit permissions for each user to a single copy of the file instead,
and you would probably want them all to have read/write access but in
fact NOT have delete permissions.

Len Sorensen

Re: what's next for the linux kernel?

[Marc Perkel]

Lennart Sorensen wrote:

>On Wed, Oct 05, 2005 at 12:44:25PM -0700, Marc Perkel wrote:
>  
>
>>What you don't get is that if you don't have rights to write to a file 
>>then you shouldn't have the right to delete the file.  Once you get past 
>>the "inside the box" Unix thinking you'll see the logic in this. So what 
>>if the process of deleting a file involves writing to it. That's not 
>>relevant.
>>    
>>
>
>When a system supports hardlinks, it IS relevant.
>
>So if I decide I want a link to a file like say /etc/group in my home
>dir (let us pretend they are on the same partition) so I make a hardlink
>to it in my home dir and end up with a file still owned by root (since I
>shouldn't be able to add me as owner to the file just by linking to it
>after all).  Should I now have to go bother the admin about deleting the
>file from my home dir if I decide that wasn't really what I wanted?  If
>I didn't have write permissions to the dir I wouldn't have been able to
>make the link in the first place, so since I made it I should be able to
>delete it, and I can with the unix way of doing things.  I still can't
>edit it anymore than I could in the original place since it linked with
>the new link to the file having the excact same permissions as the
>original.  Only someone like root can go chance the owner of a hardlink
>to someone else and start setting up some interesting file permissions
>using multiple hardlinks to one file.
>
>I suspect you can't do that on netware, so you would have to add
>explicit permissions for each user to a single copy of the file instead,
>and you would probably want them all to have read/write access but in
>fact NOT have delete permissions.
>
>Len Sorensen
>  
>
What you don't understand is that Netware's permissions mechanish is 
totally different that Linux. A hard link in Netware wouldn't inherit 
rights the way Linux does. So the user would have rights to their hard 
link to delete that link without having rights to unlink the file.

This is an important concept so pay attention. Linux stores all the 
permission to a file with that file entry. Netware doesn't. Netware 
calculates effective rights from the parent directories and it is all 
inherited unless files or directoies are explicitly set differently. So 
if files are added to other people folders then those people get rights 
to it automatically without having to go to the second step of changing 
the file's permissions.


-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 01:05:30PM -0700, Marc Perkel wrote:
> What you don't understand is that Netware's permissions mechanish is 
> totally different that Linux. A hard link in Netware wouldn't inherit 
> rights the way Linux does. So the user would have rights to their hard 
> link to delete that link without having rights to unlink the file.
> 
> This is an important concept so pay attention. Linux stores all the 
> permission to a file with that file entry. Netware doesn't. Netware 
> calculates effective rights from the parent directories and it is all 
> inherited unless files or directoies are explicitly set differently. So 
> if files are added to other people folders then those people get rights 
> to it automatically without having to go to the second step of changing 
> the file's permissions.

So if you were to moint a partition on /mnt, does the mounted thing now
have to inherit permissions from / and /mnt or from it's own root or
what?

If you chroot something, does it have access to checking permissions
past it's 'virtual' root?

Can users make hardlinks themselves on netware or does an admin have to
do it?

It seems rather inefficient to have to read 8 directories worth of
permissions and calculate them together if you have a file 8 directories
deep compared to doing a single read and compare against a 16bit value +
uid/gid check (ACL of course makes this more complex, but still only
involves reading permissions in one place since inherited permissions
are for create time, not access time).

Unix offers features netware doesn't and some of them require
permissions to work a certain way.

Len Sorensen

Re: what's next for the linux kernel?

[Horst von Brand]

Marc Perkel <marc@perkel.com> wrote:

[...]

> What you don't understand is that Netware's permissions mechanish is
> totally different that Linux. A hard link in Netware wouldn't inherit
> rights the way Linux does. So the user would have rights to their hard
> link to delete that link without having rights to unlink the file.

OK, so a "hard link" isn't (because it has separate permissions than the
original). Sorry, watered-down symlinks don't cut it. Or just by linking
the file into my place I now have rights to modify it? The later idea makes
my skin try to crawl away...

> This is an important concept so pay attention. Linux stores all the
> permission to a file with that file entry.

You are completely right: This is an extremely central concept to
everything Unix.

>                                            Netware doesn't. Netware
> calculates effective rights from the parent directories and it is all
> inherited unless files or directoies are explicitly set
> differently. So if files are added to other people folders then those
> people get rights to it automatically without having to go to the
> second step of changing the file's permissions.

Which is a very clear explanation of how broken it all is. No wonder
NetWare is no more. Files whose persmissions change depending on which way
you look at them is a nightmare. Sure, you /can/ manage that for small(ish)
setups by brute force, but it soon has to break down.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[Marc Perkel]

Horst von Brand wrote:

>Marc Perkel <marc@perkel.com> wrote:
>
>[...]
>
>  
>
>>What you don't understand is that Netware's permissions mechanish is
>>totally different that Linux. A hard link in Netware wouldn't inherit
>>rights the way Linux does. So the user would have rights to their hard
>>link to delete that link without having rights to unlink the file.
>>    
>>
>
>OK, so a "hard link" isn't (because it has separate permissions than the
>original). Sorry, watered-down symlinks don't cut it. Or just by linking
>the file into my place I now have rights to modify it? The later idea makes
>my skin try to crawl away...
>
>  
>
>>This is an important concept so pay attention. Linux stores all the
>>permission to a file with that file entry.
>>    
>>
>
>You are completely right: This is an extremely central concept to
>everything Unix.
>
>  
>
>>                                           Netware doesn't. Netware
>>calculates effective rights from the parent directories and it is all
>>inherited unless files or directoies are explicitly set
>>differently. So if files are added to other people folders then those
>>people get rights to it automatically without having to go to the
>>second step of changing the file's permissions.
>>    
>>
>
>Which is a very clear explanation of how broken it all is. No wonder
>NetWare is no more. Files whose persmissions change depending on which way
>you look at them is a nightmare. Sure, you /can/ manage that for small(ish)
>setups by brute force, but it soon has to break down.
>  
>
If you all think Netware is no more you are under an interesting 
illusion. Linux being cheap has cut into the little server market - but 
if you have thousands of servers all running off the same shared 
permissions systems - you just aren't going to do that off of Linux.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Please STOP ! [was: what's next for the linux kernel?]

[Willy Tarreau]

[Cc: list purged to save people time]

On Wed, Oct 05, 2005 at 08:50:54PM -0700, Marc Perkel wrote:
> If you all think Netware is no more you are under an interesting 
> illusion. Linux being cheap has cut into the little server market - but 
> if you have thousands of servers all running off the same shared 
> permissions systems - you just aren't going to do that off of Linux.

Please will you move those boring threads to another mailing list or
even to usenet ? This is LKML, we're only the 6th of the month and
there are already 1250 messages, 200 of which come from this thread,
and many others coming from other long off-topic threads. 20% noise
is too high and disturbting. It becomes difficult to find someone
talking about subjects related to kernel development !

Thanks in advance
Willy

(NB: I'm not interested in your reply, so please don't Cc: me)

[VERY-OT SCOX Crap] [was: what's next for the linux kernel?]

[jmerkey]

Willy Tarreau wrote:

>[Cc: list purged to save people time]
>
>On Wed, Oct 05, 2005 at 08:50:54PM -0700, Marc Perkel wrote:
>  
>
>>If you all think Netware is no more you are under an interesting 
>>illusion. Linux being cheap has cut into the little server market - but 
>>if you have thousands of servers all running off the same shared 
>>permissions systems - you just aren't going to do that off of Linux.
>>    
>>

NetWhere? did you say. Oh yeah, that broken down piece of sh_t I rewrote 
10 years ago?
The NetWhere-did-its-market-share-go operating system is dead.

>
>Please will you move those boring threads to another mailing list or
>even to usenet ? This is LKML, we're only the 6th of the month and
>there are already 1250 messages, 200 of which come from this thread,
>and many others coming from other long off-topic threads. 20% noise
>is too high and disturbting. It becomes difficult to find someone
>talking about subjects rouelated to kernel development !
>
>  
>

You should move it to the "entertainment and trolling" section the Linux 
Community. Just add the
word "merkey" somewhere in the body of the text and forward it to the 
SCOX message board.
The thread will go on for days, and George Bush, Jimbo Wales, and even 
Sollog will probably join
the discussion.

:-)

>Thanks in advance
>Willy
>
>(NB: I'm not interested in your reply, so please don't Cc: me)
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Wed, 05 Oct 2005 12:44:25 PDT, Marc Perkel said:
> What you don't get is that if you don't have rights to write to a file 
> then you shouldn't have the right to delete the file.  Once you get past 
> the "inside the box" Unix thinking you'll see the logic in this. So what 
> if the process of deleting a file involves writing to it. That's not 
> relevant.

Oddly enough, killfiles are *also* based on the concept of removing something
without the need or ability to look at the contents....

*plonk*

(Real-world analogy - if you've *ever* thrown away a box of anything without
opening the box, you understand that you don't need to be able to modify the
box to discard it.  The space enclosed by your domicile is basically read/write,
the box can be read-only and still end up in the dumpster.  On the other hand,
discarding the back staircase is a lot harder, as the frame of the house is
basically read-only, and you need to make it read-write in order to remove the
attachment points of the staircase.  Any Joe Sixpack carpenter understands this
to be entirely intuitive...)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Dave Neuer]

On 10/5/05, Marc Perkel <marc@perkel.com> wrote:
>
> What you don't get is that if you don't have rights to write to a file
> then you shouldn't have the right to delete the file.  Once you get past
> the "inside the box" Unix thinking you'll see the logic in this. So what
> if the process of deleting a file involves writing to it. That's not
> relevant.
>

No. Listen to what other people are saying. If you have the right to
_create_ the file, you have the right to _delete_ the file. In one
case you are changing the state of where something is stored --
leaving your report on a desk. Changing the contents of the report is
different.

You may not like this paradigm, but it is the Unix/POSIX paradigm and
Linux is not going to suddenly ditch it because it doesn't fit the way
you think permissions should work. As others (and you yourself) have
pointed out, there are other mechanisms in Linux to build additional
policy layers above the basic FS semantics. If you care this much
about the issue, write your own filesystem w/ its own enhanced
permissions scheme, but stop asking the Linux development community to
morph its _Unix-like_ operating system into something else and
insisting everyone who doesn't share your subjective point of view is
ignorant.

Dave

Re: what's next for the linux kernel?

[Bodo Eggert]

On Wed, 5 Oct 2005, Marc Perkel wrote:

> What you don't get is that if you don't have rights to write to a file 
> then you shouldn't have the right to delete the file.

In unix, nobody but the kernel has the right to *delete* a file. Therefore 
nobody can delete a file without write permission.

Files are deleted if the last reference is gone. If you play a music file
and unlink it while it's playing, it won't be deleted untill the player
closes the file, since an open filehandle is a reference.


If you like, you can think of it as a kind of instant garbage collection:

Files are the objects referenced by these lists, and if you own 
the object, you can change it. However, as long as there is a reference, 
you can't destroy it, since this would invalidate all references.
Instead, you must remove all references.

Directories are lists of references, and these lists are independent from
the referenced file-objects. If you own the list, you can change it by 
adding or removing files. You can even link files not owned or accessable 
by you:

7eggert@be1:~/tmp > ls -l /tmp/foo/foo
----------    1 7eggert_b users           0 2005-10-05 22:32 /tmp/foo/foo
7eggert@be1:~/tmp > ln /tmp/foo/foo .
7eggert@be1:~/tmp > ls -l
total 0
----------    2 7eggert_b users           0 2005-10-05 22:32 foo
<snip>

Do you notice the link count in the second column?

Let's remove a link:

<snip>
*switch*
7eggert_b@be1:/tmp/foo> rm foo
rm: remove write-protected regular empty file `foo'? y
*switch*
7eggert@be1:~/tmp > ls -l
total 0
----------    1 7eggert_b users           0 2005-10-05 22:32 foo
<snip>

As you can see, each directory-owner can independantly unlink the file.

BTW: The owner can change the permissions on the linked file anytime, so
even if you couldn't link non-accessable files, you could end up with
entries in your private directory you could neither access nor delete.


I can also open the file as 7eggert_b, delete it from another tty and 
still access it's contents:

<snip>
7eggert_b@be1:/tmp/foo> cat > foo
as
*switch*
be1:/home/7eggert/tmp # rm /tmp/foo/foo
be1:/home/7eggert/tmp # ll /proc/4820/fd/1
l-wx------    1 7eggert_b users          64 Oct  5 22:43 /proc/4820/fd/1 
-> /tmp/foo/foo (deleted)
be1:/home/7eggert/tmp # cat /proc/4820/fd/1
as
*switch*
df
be1:/home/7eggert/tmp # cat /proc/4820/fd/1
as
df
<snip>

As you can see, the directory entry is deleted, but the file is still
there. However, making a hard link from a /proc/pid/fd entry is not (yet?)
possible: "ln: creating hard link `baz' to `/proc/4927/fd/1':  Invalid
cross-device link".


-- 
Fun things to slip into your budget
Not in a budget, but in an annual report:
An employee stole 500,000+. They accounted for it on the annual report as
'involountary employee relations expense'

'Undeleting' an open file

[Giuseppe Bilotta]

On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:

> Files are deleted if the last reference is gone. If you play a music file
> and unlink it while it's playing, it won't be deleted untill the player
> closes the file, since an open filehandle is a reference.

BTW, I've always wondered: is there a way to un-unlink such a file?

-- 
Giuseppe "Oblomov" Bilotta

"I weep for our generation" -- Charlie Brown

Re: 'Undeleting' an open file

[Bernd Petrovitsch]

On Fri, 2005-10-07 at 16:14 +0200, Giuseppe Bilotta wrote:
> On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
> 
> > Files are deleted if the last reference is gone. If you play a music file
> > and unlink it while it's playing, it won't be deleted untill the player
> > closes the file, since an open filehandle is a reference.
> 
> BTW, I've always wondered: is there a way to un-unlink such a file?

Legally not.
You could turn off the power and do a fsck on a not-journalling
filesystem. But even then it is not sure ...

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: 'Undeleting' an open file

[Jose Celestino]

Words by Giuseppe Bilotta [Fri, Oct 07, 2005 at 04:14:27PM +0200]:
> On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
> 
> > Files are deleted if the last reference is gone. If you play a music file
> > and unlink it while it's playing, it won't be deleted untill the player
> > closes the file, since an open filehandle is a reference.
> 
> BTW, I've always wondered: is there a way to un-unlink such a file?
> 

Well, you can go to /proc/$PID_OF_PROCESSING_OPENING_THE_FILE/fd/

do an ls -la, identify the fd of the file deleted but still open
and do a cp $FD ~/my_deleted_file

Example:

[japc@morgoth:/proc/3743/fd]$ ls -al
total 4
dr-x------  2 japc japc  0 Oct  7 15:25 .
dr-xr-xr-x  4 japc japc  0 Oct  7 15:23 ..
lr-x------  1 japc japc 64 Oct  7 15:25 0 -> /dev/null
l-wx------  1 japc japc 64 Oct  7 15:25 1 -> /home/japc/.xsession-errors
l-wx------  1 japc japc 64 Oct  7 15:25 2 -> /home/japc/.xsession-errors
lr-x------  1 japc japc 64 Oct  7 15:25 255 -> /usr/bin/startkde
[japc@morgoth:/proc/3743/fd]$ cp 255 ~/startkde


-- 
Jose Celestino | http://xpto.org/~japc/files/japc-pgpkey.asc
----------------------------------------------------------------
"Thus the metric system did not really catch on in the States, unless you count
the increasing popularity of the nine-millimeter bullet." -- Dave Barry

Re: 'Undeleting' an open file

[Ian Campbell]

On Fri, 2005-10-07 at 16:14 +0200, Giuseppe Bilotta wrote:
> On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
> 
> > Files are deleted if the last reference is gone. If you play a music file
> > and unlink it while it's playing, it won't be deleted untill the player
> > closes the file, since an open filehandle is a reference.
> 
> BTW, I've always wondered: is there a way to un-unlink such a file?

Access via /proc/PID/fd/* seems to work:

$ echo "Hello World" > testing
$ exec 10>>testing
$ rm testing
$ ls -l /proc/self/fd/
total 5
lrwx------  1 icampbell icampbell 64 Oct  7 15:28 0 -> /dev/pts/9
lrwx------  1 icampbell icampbell 64 Oct  7 15:28 1 -> /dev/pts/9
l-wx------  1 icampbell icampbell 64 Oct  7 15:28 10
-> /home/icampbell/testing (deleted)
lrwx------  1 icampbell icampbell 64 Oct  7 15:28 2 -> /dev/pts/9
lr-x------  1 icampbell icampbell 64 Oct  7 15:28 3 -> /proc/31390/fd/
$ cat /proc/self/fd/10
Hello World
$

Ian.
-- 
Ian Campbell
Current Noise: Rotting Christ - In Domine Sathana

I may not be totally perfect, but parts of me are excellent.
		-- Ashleigh Brilliant

Re: 'Undeleting' an open file

[Bernd Petrovitsch]

On Fri, 2005-10-07 at 15:30 +0100, Ian Campbell wrote:
> On Fri, 2005-10-07 at 16:14 +0200, Giuseppe Bilotta wrote:
> > On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
> > 
> > > Files are deleted if the last reference is gone. If you play a music file
> > > and unlink it while it's playing, it won't be deleted untill the player
> > > closes the file, since an open filehandle is a reference.
> > 
> > BTW, I've always wondered: is there a way to un-unlink such a file?
> 
> Access via /proc/PID/fd/* seems to work:
> 
> $ echo "Hello World" > testing
> $ exec 10>>testing
> $ rm testing
> $ ls -l /proc/self/fd/
> total 5
> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 0 -> /dev/pts/9
> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 1 -> /dev/pts/9
> l-wx------  1 icampbell icampbell 64 Oct  7 15:28 10
> -> /home/icampbell/testing (deleted)
> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 2 -> /dev/pts/9
> lr-x------  1 icampbell icampbell 64 Oct  7 15:28 3 -> /proc/31390/fd/
> $ cat /proc/self/fd/10
> Hello World
> $

Did you try linking it?

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: 'Undeleting' an open file

[Andreas Schwab]

Bernd Petrovitsch <bernd@firmix.at> writes:

> On Fri, 2005-10-07 at 15:30 +0100, Ian Campbell wrote:
>> Access via /proc/PID/fd/* seems to work:
>> 
>> $ echo "Hello World" > testing
>> $ exec 10>>testing
>> $ rm testing
>> $ ls -l /proc/self/fd/
>> total 5
>> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 0 -> /dev/pts/9
>> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 1 -> /dev/pts/9
>> l-wx------  1 icampbell icampbell 64 Oct  7 15:28 10
>> -> /home/icampbell/testing (deleted)
>> lrwx------  1 icampbell icampbell 64 Oct  7 15:28 2 -> /dev/pts/9
>> lr-x------  1 icampbell icampbell 64 Oct  7 15:28 3 -> /proc/31390/fd/
>> $ cat /proc/self/fd/10
>> Hello World
>> $
>
> Did you try linking it?

link(2) does not follow symlinks, and there is no flink(2).

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: 'Undeleting' an open file

[Alex Riesen]

On 10/7/05, Bernd Petrovitsch <bernd@firmix.at> wrote:
> > > > Files are deleted if the last reference is gone. If you play a music file
> > > > and unlink it while it's playing, it won't be deleted untill the player
> > > > closes the file, since an open filehandle is a reference.
> > > BTW, I've always wondered: is there a way to un-unlink such a file?
> > Access via /proc/PID/fd/* seems to work:
> Did you try linking it?
>

ln: creating hard link `testfile2' to `/proc/14282/fd/3': Invalid
cross-device link
Pity :)
"cp" works, btw.

Re: 'Undeleting' an open file

[Bernd Petrovitsch]

On Fri, 2005-10-07 at 17:14 +0200, Alex Riesen wrote:
> On 10/7/05, Bernd Petrovitsch <bernd@firmix.at> wrote:
> > > > > Files are deleted if the last reference is gone. If you play a music file
> > > > > and unlink it while it's playing, it won't be deleted untill the player
> > > > > closes the file, since an open filehandle is a reference.
> > > > BTW, I've always wondered: is there a way to un-unlink such a file?
> > > Access via /proc/PID/fd/* seems to work:
> > Did you try linking it?
> >
> 
> ln: creating hard link `testfile2' to `/proc/14282/fd/3': Invalid
> cross-device link
> Pity :)
> "cp" works, btw.

Yup.
<anal>But then it is not un-unlinking and you get another file.</anal>

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: 'Undeleting' an open file

[Miquel van Smoorenburg]

In article <8qo997np4h6n.1ihs13ptrx2y2.dlg@40tude.net>,
Giuseppe Bilotta  <bilotta78@hotpop.com> wrote:
>On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
>
>> Files are deleted if the last reference is gone. If you play a music file
>> and unlink it while it's playing, it won't be deleted untill the player
>> closes the file, since an open filehandle is a reference.
>
>BTW, I've always wondered: is there a way to un-unlink such a file?

Around 2.4.15 we had the same discussion here - see for example
http://www.ussg.iu.edu/hypermail/linux/kernel/0201.2/0881.html
(but be sure to read the whole thread).

Mike.

Re: 'Undeleting' an open file

[Giuseppe Bilotta]

On Fri, 7 Oct 2005 14:30:37 +0000 (UTC), Miquel van Smoorenburg wrote:

> In article <8qo997np4h6n.1ihs13ptrx2y2.dlg@40tude.net>,
> Giuseppe Bilotta  <bilotta78@hotpop.com> wrote:
>>On Wed, 5 Oct 2005 23:05:34 +0200 (CEST), Bodo Eggert wrote:
>>
>>> Files are deleted if the last reference is gone. If you play a music file
>>> and unlink it while it's playing, it won't be deleted untill the player
>>> closes the file, since an open filehandle is a reference.
>>
>>BTW, I've always wondered: is there a way to un-unlink such a file?
> 
> Around 2.4.15 we had the same discussion here - see for example
> http://www.ussg.iu.edu/hypermail/linux/kernel/0201.2/0881.html
> (but be sure to read the whole thread).

A very interesting read, thank you for the link.

-- 
Giuseppe "Oblomov" Bilotta

Axiom I of the Giuseppe Bilotta
theory of IT:
Anything is better than MS

Re: what's next for the linux kernel?

[Steven Rostedt]

On Wed, 5 Oct 2005, Marc Perkel wrote:
> What you don't get is that if you don't have rights to write to a file
> then you shouldn't have the right to delete the file.  Once you get past
> the "inside the box" Unix thinking you'll see the logic in this. So what
> if the process of deleting a file involves writing to it. That's not
> relevant.

Marc, Get off of it!

In 1987 I first started using Unix. Before that I worked mostly on DOS.
At first the delete w/o ownership shocked me, but then I changed my POV
thinking and found that unlinking from a directory only took write access
to the directory and it made sense to me.  This was all before I began to
"think inside the Unix box".  In fact, it was when I was "thinking inside
the DOS box".

God I was 19 at the time and I easily got a clue. You are stuck on your
own POV and you cant seem to see anything any other way.  So stop trying
to convince us.  We can make up our own minds and already have.

-- Steve

Re: what's next for the linux kernel?

[Bernd Petrovitsch]

On Wed, 2005-10-05 at 07:48 -0700, Marc Perkel wrote:
[...]
> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 

One that allows you to remove read-only files.

> only the Unix community doesn't get it.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: what's next for the linux kernel?

[Joe Bob Spamtest]

Marc Perkel wrote:
>> It would be an incredibly idiotic system that auto hides files just
>> because you can't use them.  We have ways to hide files in user space
>> for the convinience of users.  It would be too inconvinient for
>> applications if the OS hid files on us.
>>
>> Len Sorensen
> 
> What is incredibly idiotic is a file system that allws you to delete 
> files that you have no write access to. That is stupid beyond belief and 
> only the Unix community doesn't get it.

A: You're stupid!
B: No, You're stupid!
A: My dad can beat up your dad!
B: No, my dad can beat up yours!
A: My penis is bigger than yours!
B: No, my penis is bigger!
A: (ad infinium)

Look, for those of you who love their beloved netware, go back and use 
it. You don't like the way the unix filesystems work because you don't 
UNDERSTAND how they work. You're talking about a bunch of special-case 
stuff here -- POSIX ACLs and SELinux are your answer. Use them. But for 
christ's sake, STOP BITCHING ABOUT NETWARE.

Re: what's next for the linux kernel?

[linux-os (Dick Johnson)]

On Wed, 5 Oct 2005, Lennart Sorensen wrote:

> On Wed, Oct 05, 2005 at 07:41:21AM -0700, Marc Perkel wrote:
>> If you were going to do it right here's what you would do:
>>
>> People who had files in /tmp would have no rights at all to other users
>> /tmp files.
>> Listing the dirtectory would only display the files you had some access
>> to. If you have no rights you don't even see that the file is there.
>> The effect would be like giving people their own tmp directories.
>
> Except it still wouldn't be able to go: Does file xyz exist?  If not,
> create file xyz.  If someone else had xyz that you didn't see, you would
> still not be able to create it.  So what is the point of NOT showing it
> other than to make it much harder to avoid conflicting names?
>
> if you want to not see files that you have no rights to, filter it in
> your user space application when it matters, and let user space see the
> files when they need to in order to avoid name conflicts.
>
> It would be an incredibly idiotic system that auto hides files just
> because you can't use them.  We have ways to hide files in user space
> for the convinience of users.  It would be too inconvinient for
> applications if the OS hid files on us.
>
> Len Sorensen
> -

Also it has nothing at all to do with the kernel. It's what `ls`
or some other directory-reading program provides for the user.
People often forget that PATH, `pwd`, etc., are just filter
components!

When you `cd` to somewhere, your location hasn't changed at
all!

Without involving the kernel, one can make any kind of filter
to cause any sort of display that you want.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 10:52:59AM -0400, linux-os (Dick Johnson) wrote:
> Also it has nothing at all to do with the kernel. It's what `ls`
> or some other directory-reading program provides for the user.
> People often forget that PATH, `pwd`, etc., are just filter
> components!
> 
> When you `cd` to somewhere, your location hasn't changed at
> all!

So what does bash do that makes the new location 'busy' when you cd to
it such that you can't unmount it?

> Without involving the kernel, one can make any kind of filter
> to cause any sort of display that you want.

An it certainly is something that should be done in user space.

Len Sorensen

Re: what's next for the linux kernel?

[linux-os (Dick Johnson)]

On Wed, 5 Oct 2005, Lennart Sorensen wrote:

> On Wed, Oct 05, 2005 at 10:52:59AM -0400, linux-os (Dick Johnson) wrote:
>> Also it has nothing at all to do with the kernel. It's what `ls`
>> or some other directory-reading program provides for the user.
>> People often forget that PATH, `pwd`, etc., are just filter
>> components!
>>
>> When you `cd` to somewhere, your location hasn't changed at
>> all!
>
> So what does bash do that makes the new location 'busy' when you cd to
> it such that you can't unmount it?


Well it doesn't make a file-system location busy! It's only files
that are open-for-write that prevent a file-system from being un-mounted!

You can properly shut down a system with the following commands from
a dumb terminal (ctrl-ALT F1,F2, etc).

kill -TERM -1	# Kill everybody but me and 'init'
sleep 1		# Wait a bit
kill -9 -1	# Really kick the hangers-on
sleep 1		# Wait again
umount -a	# Umount all file systems

After you execute `umount -a`, you can still read the file-system
because `umount` only made it R/O.

`>foo`
shows that the file-system is R/O, you can hit the reset or power
switch now.

Certain distros create a file in the top directory that is supposed
to tell startup that the system was not properly shut down, "/.autofsck",
if you deleted that before the above sequence, the machine can
be restarted with no informational error messages about the shutdown.

`cd` executes function-code 12 which makes all opens() start from the
input string "path" if it doesn't have a full path. It's just a filter.
Same for opendir() if a directory listing is to be obtained.

>
>> Without involving the kernel, one can make any kind of filter
>> to cause any sort of display that you want.
>
> An it certainly is something that should be done in user space.
>

Could be done from user-space but opening an ordinary file
would require that the full path be obtained from somewhere
because the kernel wouldn't "know" where to create it if the
full path wasn't part of the open. `cd` is a kernel-call that
conveniently stores the part of the path-name that you don't
want to have to repeat for every open.

> Len Sorensen
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 07:41:21AM -0700, Marc Perkel wrote:
> If you were going to do it right here's what you would do:
> 
> People who had files in /tmp would have no rights at all to other users 
> /tmp files.
> Listing the dirtectory would only display the files you had some access 
> to. If you have no rights you don't even see that the file is there.
> The effect would be like giving people their own tmp directories.
 
  ahh, *sigh*, i remember the days.

  in 1989 i looked in /tmp on our sunos 4.1.3 server at
  imperial, which was running a bit slow, went "eek, that's
  a lot of files in /tmp" and did am rm -fr /tmp.

  a few minutes later the sysadmins quite literally stormed in.

  apparently the printer queue temp files were stored in /tmp and 100
  third year students were all trying to print out their course-work,
  last minute.

  oops.

  yes, imperial college third year theory of computing students of
  1987-1990, it was me.

  l.

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 04:24:47PM +0100, Luke Kenneth Casson Leighton wrote:
>   ahh, *sigh*, i remember the days.
> 
>   in 1989 i looked in /tmp on our sunos 4.1.3 server at
>   imperial, which was running a bit slow, went "eek, that's
>   a lot of files in /tmp" and did am rm -fr /tmp.

Why would /tmp allow you to delete files there you didn't own unless you
were root?  Why would someone with root blindly delete things they
didn't know what were?

>   a few minutes later the sysadmins quite literally stormed in.

And promptly removed root access from the person that wasn't qualified
to have it in the first place? :)

>   apparently the printer queue temp files were stored in /tmp and 100
>   third year students were all trying to print out their course-work,
>   last minute.
> 
And why would the printer queue use /tmp in the first place?

>   oops.
> 
>   yes, imperial college third year theory of computing students of
>   1987-1990, it was me.

Did they ever let you have root again?

Len Sorensen

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 11:30:06AM -0400, Lennart Sorensen wrote:
> On Wed, Oct 05, 2005 at 04:24:47PM +0100, Luke Kenneth Casson Leighton wrote:
> >   ahh, *sigh*, i remember the days.
> > 
> >   in 1989 i looked in /tmp on our sunos 4.1.3 server at
> >   imperial, which was running a bit slow, went "eek, that's
> >   a lot of files in /tmp" and did am rm -fr /tmp.
> 
> Why would /tmp allow you to delete files there you didn't own unless you
> were root?  

 i have no idea.  as a user, i just did rm -fr /tmp/* (sorry - not
 rm -fr /tmp) and it worked.

 as a user.

 not root.

> >   a few minutes later the sysadmins quite literally stormed in.
> 
> And promptly removed root access from the person that wasn't qualified
> to have it in the first place? :)
 
 they weren't dumb enough to give it to me.

> >   apparently the printer queue temp files were stored in /tmp and 100
> >   third year students were all trying to print out their course-work,
> >   last minute.
> > 
> And why would the printer queue use /tmp in the first place?
 
 ahh, that would answer the implicit question as to why they
 jumped up and down at me rather than frog-marched me off campus.

> >   oops.
> > 
> >   yes, imperial college third year theory of computing students of
> >   1987-1990, it was me.
> 
> Did they ever let you have root again?

 i was a student there.  they didn't let _anyone_ like me have root.
 
 someone got into trouble for even demonstrating a security
 vulnerability.

 l.

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 04:42:26PM +0100, Luke Kenneth Casson Leighton wrote:
>  i have no idea.  as a user, i just did rm -fr /tmp/* (sorry - not
>  rm -fr /tmp) and it worked.
> 
>  as a user.
> 
>  not root.

Then some admin didn't qualify for root having apparently removed the t
bit from /tmp making it a world writeable dir.  Ouch.

>  they weren't dumb enough to give it to me.

But they made /tmp world writeable it seems.  Impresive. :)

>  ahh, that would answer the implicit question as to why they
>  jumped up and down at me rather than frog-marched me off campus.

Yep.  What you did should have been prevented by the system.  So the
system was misconfigured.

>  i was a student there.  they didn't let _anyone_ like me have root.
>  
>  someone got into trouble for even demonstrating a security
>  vulnerability.

Well this one sounds more liek a major misconfiguration than a security
problem.  Well allowing people to mess with temp could be seen as a
security problem but only until the permissions were fixed back to what
they would have originally been when the system was installed.

Len Sorensen

Re: what's next for the linux kernel?

[Ragnar Hojland Espinosa]

On Wed, Oct 05, 2005 at 11:55:16AM -0400, Lennart Sorensen wrote:
> On Wed, Oct 05, 2005 at 04:42:26PM +0100, Luke Kenneth Casson Leighton wrote:
> >  i have no idea.  as a user, i just did rm -fr /tmp/* (sorry - not
> >  rm -fr /tmp) and it worked.
> > 
> >  as a user.
> > 
> >  not root.
> 
> Then some admin didn't qualify for root having apparently removed the t
> bit from /tmp making it a world writeable dir.  Ouch.
> 
> >  they weren't dumb enough to give it to me.
> 
> But they made /tmp world writeable it seems.  Impresive. :)

Silly accidents like that happen.  A lazy tarballer in action:

# ls -ld foo
drwxr-xr-x  2 root   root  48 Oct  6 17:34 foo
# cd foo
# tar cf ../foo.tar .

And too sleepy root who blindly untars to /tmp

# ls -ld tmp
drwxrwxrwt  2 root   root  48 Oct  6 17:34 tmp
# tar xf foo.tar
# ls -ld tmp
drwxr-xr-x  2 root   root     72 Oct  6 17:36 tmp

woops.

-- 
Ragnar Hojland - Project Manager
Linalco "Specialists in Linux and Free Software"
http://www.linalco.com  Tel: +34-91-4561700

Re: what's next for the linux kernel?

[David Leimbach]

On 10/5/05, Nix <nix@esperi.org.uk> wrote:
> On 4 Oct 2005, Bodo Eggert stated:
> > BTW: YANI: That about a tmpfs where all-numerical entries can only be
> > created by the corresponding UID? This would provide a secure, private
> > tmp directory to each user without the possibility of races and denial-of-
> > service attacks. Maybe it should be controlled by a mount flag.
>
> Wouldn't it be less kludgy to just use the existing private namespace
> stuff to provide each user with its own /tmp? (Or each user's session,
> rather, which is probably much easier, as that corresponds precisely to
> one process tree).
>

It would if the rest of the system really enforced this "privacy".  In
plan 9 /tmp is really a bind to /usr/$user/tmp.  And if you launch
something like "ramfs" [a userland 9P server] it binds a ram disk
device over /tmp by default unless you tell it otherwise, then you
have a ram-backed directory only for the current process and its
children in /tmp.  This is useful for pulling things out of the
encrypted storage like factotum keys [sort of like a keyring for all
factotum based authentication including 9P mounts and even ssh
connections that use no ssh-keys].  When your process goes away so
does the decrypted keyfile, pretty nice.

Back on topic...

The problem with private namespaces on Linux is that they really
aren't so much.  mount will update /etc/mtab for all to see and even
/proc/<pid>/mounts is world readable [though it doesn't give useful
bind information anyway on linux... just the disk device it appears].

On one hand you've got very specific information in mtab about all the
binding that's been done and on the other hand you've got not so
useful information on a per-process basis in /proc.

I think private namespaces could actually be made more-so but the rest
of the system has to cooperate and I doubt that I have the energy to
do the evangelism and requisite proofs of concept for Linux.  It's far
easier for me to just use Plan 9 and Inferno instead of trying to
assimilate Linux, even though I think I'd prefer Linux if it were more
like the former two.

Maybe it's time for another BSD fork? [*runs away from the disapproval*]

Dave

Re: what's next for the linux kernel?

[Bodo Eggert]

On Wed, 5 Oct 2005, David Leimbach wrote:

[snip quotes]

> It would if the rest of the system really enforced this "privacy".  In
> plan 9 /tmp is really a bind to /usr/$user/tmp.  And if you launch
> something like "ramfs" [a userland 9P server] it binds a ram disk
> device over /tmp by default unless you tell it otherwise, then you
> have a ram-backed directory only for the current process and its
> children in /tmp.
[...]

> This is useful for pulling things out of the
> encrypted storage like factotum keys [sort of like a keyring for all
> factotum based authentication including 9P mounts and even ssh
> connections that use no ssh-keys].  When your process goes away so
> does the decrypted keyfile, pretty nice.

You'd usurally just create+open a file and erase it without closing it.
The only access to this file is by using the file descriptor (or, off 
cause, /proc/pid/fd/num). If the last reference to this file, the running 
process, is gone, so is the file.

> Back on topic...
> 
> The problem with private namespaces on Linux is that they really
> aren't so much.  mount will update /etc/mtab for all to see and even

Userspace problem.-)

> /proc/<pid>/mounts is world readable [though it doesn't give useful
> bind information anyway on linux... just the disk device it appears].

There was some proc privacy patch some time ago. It was argued about 
because some sites want peer review on system usage. I lost track 
if it was included.

> I think private namespaces could actually be made more-so but the rest
> of the system has to cooperate and I doubt that I have the energy to
> do the evangelism and requisite proofs of concept for Linux.  It's far
> easier for me to just use Plan 9 and Inferno instead of trying to
> assimilate Linux, even though I think I'd prefer Linux if it were more
> like the former two.

The plan is:

1) make namespaces joinable
2) ???
3) profit

No, that's wrong. The plan is (should be?):

1) make namespaces joinable in a sane way
2) wait for the shared subtree patch
3) make pam join the per-user-namespace
4) make pam automount tmpfs on the private /tmp

-- 
Top 100 things you don't want the sysadmin to say:
44. System coming down in 0 min....

Re: what's next for the linux kernel?

[David Leimbach]

ss, is gone, so is the file.
>
> > Back on topic...
> >
> > The problem with private namespaces on Linux is that they really
> > aren't so much.  mount will update /etc/mtab for all to see and even
>
> Userspace problem.-)

Sure is, which is why I think it's easier to fork a BSD to make it do
what someone wants than to roll my own linux distribution :-).  
Perhaps that's a problem of perception and not a really good
measurement of the amount of energy involved in either alternative.

Sometimes it sure seems easier to keep the userspace stuff with the kernel.

> > I think private namespaces could actually be made more-so but the rest
> > of the system has to cooperate and I doubt that I have the energy to
> > do the evangelism and requisite proofs of concept for Linux.  It's far
> > easier for me to just use Plan 9 and Inferno instead of trying to
> > assimilate Linux, even though I think I'd prefer Linux if it were more
> > like the former two.
>
> The plan is:
>
> 1) make namespaces joinable
> 2) ???
> 3) profit
>
> No, that's wrong. The plan is (should be?):
>
> 1) make namespaces joinable in a sane way
> 2) wait for the shared subtree patch
> 3) make pam join the per-user-namespace
> 4) make pam automount tmpfs on the private /tmp

I'm not sure what you mean by a joinable namespace.  I also am not
sure I want them :-).

I think of namespaces as being fundamental to the process model and
that they are inherited from the parent and new ones are created in a
sort of COW fashion [or at least have similar behavior].

You might want a session namespace instead of a joinable per-process
namespace but I think that might be a slightly different point of
view.

Dave

Re: what's next for the linux kernel?

[Nix]

On Wed, 5 Oct 2005, David Leimbach said:
>> 1) make namespaces joinable in a sane way
>> 2) wait for the shared subtree patch
>> 3) make pam join the per-user-namespace
>> 4) make pam automount tmpfs on the private /tmp
> 
> I'm not sure what you mean by a joinable namespace.  I also am not
> sure I want them :-).

They are namespaces which processes can join. Right now you can do
it by chrooting into /proc/{pid}/root, but this is, as Bodo said,
not a very sane API.

Without this, a user starting two sessions gets two namespaces,
which is profoundly counterintuitive from the user's POV.

> I think of namespaces as being fundamental to the process model and
> that they are inherited from the parent and new ones are created in a
> sort of COW fashion [or at least have similar behavior].

Yes, but you can change them too (that's what e.g. mount() is for!)

> You might want a session namespace instead of a joinable per-process
> namespace but I think that might be a slightly different point of
> view.

I think that's the idea; a filesystem holding namespaces that you're
allowed to chroot() into.

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

> > 1) make namespaces joinable in a sane way
> > 2) wait for the shared subtree patch
> > 3) make pam join the per-user-namespace
> > 4) make pam automount tmpfs on the private /tmp
 
 per-user namespaces on /tmp would solve a lot of the problems faced by
 selinux with respect to directory permissions on /tmp/.font-unix
 and /tmp/.X11-unix.
 
 there are a lot of legacy apps that no-one wants to modify to get them
 to create/read /tmp/x-windows/.X11-unix.

 oops.

Re: what's next for the linux kernel?

[grundig]

El Thu, 6 Oct 2005 00:23:30 +0100,
Luke Kenneth Casson Leighton <lkcl@lkcl.net> escribió:

>  there are a lot of legacy apps that no-one wants to modify to get them
>  to create/read /tmp/x-windows/.X11-unix.

What's the point of caring about security for a legacy app if nobody 
is going to fix it if a security problema arises?


http://packages.debian.org/unstable/admin/libpam-tmpdir 

is good enought IMO

Re: what's next for the linux kernel?

[Tomasz Kłoczko]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1074 bytes --]

On Thu, 6 Oct 2005 grundig@teleline.es wrote:

> El Thu, 6 Oct 2005 00:23:30 +0100,
> Luke Kenneth Casson Leighton <lkcl@lkcl.net> escribió:
>
>>  there are a lot of legacy apps that no-one wants to modify to get them
>>  to create/read /tmp/x-windows/.X11-unix.
>
> What's the point of caring about security for a legacy app if nobody
> is going to fix it if a security problema arises?
>
>
> http://packages.debian.org/unstable/admin/libpam-tmpdir
>
> is good enought IMO

BTW. Also it will be good say something about storing unix sockets by 
system programs (/tmp/x-windows/.X11-unix which is owned by root.root) in 
/tmp.

http://www.pathname.com/fhs/pub/fhs-2.3.html#REQUIREMENTS14
says about /var/run:
"System programs that maintain transient UNIX-domain sockets must place 
them in this directory."

kloczek
-- 
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: kloczek@rudy.mif.pg.gda.pl*

Re: what's next for the linux kernel?

[Greg Norris]

On Thu, Oct 06, 2005 at 11:53:39AM +0200, grundig@teleline.es wrote:
> What's the point of caring about security for a legacy app if nobody 
> is going to fix it if a security problema arises?
> 
> 
> http://packages.debian.org/unstable/admin/libpam-tmpdir 
> 
> is good enought IMO

That reminds me, gdm hangs (on Debian sid, anyway) when using the 
pam_tmpdir.so module.  I was meaning to file a bugreport... guess I'd 
better get moving.

Re: what's next for the linux kernel?

[Marc Perkel]

Bodo Eggert wrote:

>Marc Perkel <marc@perkel.com> wrote:
>
>[...]
>  
>
>>I'll run through a few ideas here.
>>    
>>
>
>  
>
>>Novell Netware type permissions. ACLs are a step in the right direction
>>but Linux isn't any where near where Novell was back in 1990. Linux lets
>>you - for example - to delete files that you have no read or write
>>access rights to.
>>    
>>
>
>It lets you unlink them. That's different from deleting, since the owner
>may have his/her private link to that file.
>
>Unlinking is changing the contents of a directory, and it's controlled by
>the write permission of the containing directory.
>
>  
>
There would be different rights to eack link.

>>Netware on the other hand prevents you from deleting
>>files that you can't write to and if you have no right it is as if the
>>file isn't there.
>>    
>>
>
>Imagine a /tmp directory (writable by world) with user "a" creating a file
>"foo", umask=077 and user "b" trying to do the same. User "b" will get
>'file exists' if he tries to create it, and 'file does not exist' if he
>tries to list it. He will go nuts.
>  
>
Users should have private temp directory space. Two user trying to 
create the same file in the same directory isn't going to work under any 
operating system.

>BTW: YANI: That about a tmpfs where all-numerical entries can only be
>created by the corresponding UID? This would provide a secure, private
>tmp directory to each user without the possibility of races and denial-of-
>service attacks. Maybe it should be controlled by a mount flag.
>
>  
>
>>You can't even see it in the directory. Netware also
>>has inherited permissions like Windows and Samba has and this is doing
>>it right.
>>    
>>
>
>You can't do that if you have hardlinks. However, I missed the feature of
>overruling file permissions in some special directories, e.g. anything
>put under /pub should ignore umask and be a+rX.
>  
>
You have to realize the Netware does things differently and that Linux 
limitations don't apply to Netware.

>  
>
>>File systems and individual directories should be able to be
>>flagged as casesensitive/insensitive.
>>    
>>
>
>IMHO not needed.
>  
>
But just because you don't need it doesn't mean other people don't. If 
you are running Samba pretending to be a case insensitive file system 
then this is a good feature.

Re: what's next for the linux kernel?

[Julian Blake Kongslie]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

On Wed, 05 Oct 2005 13:27:15 -0700
Marc Perkel <marc@perkel.com> wrote:
> There would be different rights to eack link.

Well, color me confused.

You appear to be saying that the permission on a file differ depending
on which link you are accessing it by. Furthermore, your stance seems to
imply that linking to a file grants either write permission or ownership
on the new link.

So, under this permission model, I could link to /etc/passwd in my
home directory, edit the link to change my UID to zero, then relogin to
the system as an administrator.

Not that I would need to, of course, because any user who owns/could
write to a directory would be able to alter any file on the entire
system. I know they're called "permission" models, but that seems
*extremely* permissive...

-- 
-Julian Blake Kongslie
<jblake@omgwallhack.org>

[-- Attachment #2: Type: application/pgp-signature, Size: 287 bytes --]

Re: what's next for the linux kernel?

[Bas Westerbaan]

You can delete a directory entry to a file if you have proper
permission to the directory.

You cannot read or write the file if the file doesn't give you permission to.

A hard link makes an additional directory entry to a certain file. You
delete the directory entry to the file, not the file.

And permissions are the same for all instances of the file.

My 2 cents.

On 10/5/05, Julian Blake Kongslie <jblake@omgwallhack.org> wrote:
> On Wed, 05 Oct 2005 13:27:15 -0700
> Marc Perkel <marc@perkel.com> wrote:
> > There would be different rights to eack link.
>
> Well, color me confused.
>
> You appear to be saying that the permission on a file differ depending
> on which link you are accessing it by. Furthermore, your stance seems to
> imply that linking to a file grants either write permission or ownership
> on the new link.
>
> So, under this permission model, I could link to /etc/passwd in my
> home directory, edit the link to change my UID to zero, then relogin to
> the system as an administrator.
>
> Not that I would need to, of course, because any user who owns/could
> write to a directory would be able to alter any file on the entire
> system. I know they're called "permission" models, but that seems
> *extremely* permissive...
>
> --
> -Julian Blake Kongslie
> <jblake@omgwallhack.org>
>
>
>



--
Bas Westerbaan
http://blog.w-nz.com/
GPG Public Keys: http://w-nz.com/keys/bas.westerbaan.asc

Re: what's next for the linux kernel?

[Julian Blake Kongslie]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

On Wed, 5 Oct 2005 22:51:32 +0200
Bas Westerbaan <bas.westerbaan@gmail.com> wrote:

> You can delete a directory entry to a file if you have proper
> permission to the directory.
> 
> You cannot read or write the file if the file doesn't give you
permission to.
> 
> A hard link makes an additional directory entry to a certain file. You
> delete the directory entry to the file, not the file.
> 
> And permissions are the same for all instances of the file.
> 
> My 2 cents.

That is the UNIX model, yes. And I think it makes perfect sense. And as
a side effect, we can delete links to files which we do not own, and
cannot write to.

Does NetWare have an equivalent of hardlinks?

-- 
-Julian Blake Kongslie
<jblake@omgwallhack.org>

[-- Attachment #2: Type: application/pgp-signature, Size: 287 bytes --]

Re: what's next for the linux kernel?

[Bodo Eggert]

Nix <nix@esperi.org.uk> wrote:
> On 4 Oct 2005, Marc Perkel announced authoritatively:

>>             Netware also has inherited permissions like Windows and
>> Samba has and this is doing it right.
> 
> s/right/wrong/
> 
> Look at a NetWare permission on some file and you can't tell what the
> effective permission is, because it depends on inherited ones as well.
> Look at an effective permission and you can't tell which parts of it
> will change if you change the inherited permission.

MS solved that part by not allowing partially defined permissions.
(At least AFAI can tell from the UI.)

> Unix has inherited permissions in one sense: you can't get at a file
> via some path if one of the directories in that path is unreadable.

NACK. ITYM non-accessable (-x), and it's only true if you can't reach it or
one of the other links from $PWD and the directories you can fchdir to
(which, admittedly, is the usural case).

> Even *that* causes regular problems,

ACK.-)

>>                                       File systems and individual
>> directories should be able to be flagged as casesensitive/insensitive.
> 
> Only if you're willing to change POSIX to include a call to check filenames
> for identity,

You'd "just" need a way to determine the canonialized form. Still an evil
masterplan.

[...]
> It would also require case-conversion and locale-handling code, probably
> including UTF-8 canoncalization code, inside the kernel. This would
> greatly increase kernel complexity for a very small reward, and lead to
> Al Viro's early death from cerebral aneurysm combined with involuntary
> projectile vomiting. This cannot be considered a good thing.

a) NLS is in the kernel, and if utf-8 filenames are supposed to be used,
   an utf-8 checker rejecting non-canonialized strings will be desirable
   to avoid binary trash in filenames or lookalike filenames. The
   conversion to the canonialized form should happen outside the kernel.

   (I know POSIX doesn't define a propper return value, but the return
    value used for VFAT works and it's better than dealing with
    $'M\x{0430}kefile' looking like Makefile)

b) ACK, I don't think caseless handling of filenames is a good thing,
   it would needlessly bloat the kernel by opening a can of worms.
   E.g. 'ß' would be converted to 'SS'[0] in German or 'B' in greek.


[0] or, if you like and bend the standard, to 'SZ' if the word with 'SS'
    would clash with another word.

> Now /etc/mtab, *that* is an abomination, and a small kernel improvement
> (allowing arbitrary flag strings to be passed by mount into the kernel
> solely for display in the appropriate /proc/mounts field) could
> eliminate it and replace it entirely with /proc/mounts.

What about making all fs ignore the
-omount="$tool:foo:bar;$tool2=baz:barf..." parameter?

This is a cruel hack, but it will be backward compatible.
(If your hammer is big enough, the problem may turn out to be a nail.)

-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

Re: what's next for the linux kernel?

[Nix]

On Wed, 05 Oct 2005, Bodo Eggert suggested tentatively:
> Nix <nix@esperi.org.uk> wrote:
>> On 4 Oct 2005, Marc Perkel announced authoritatively:
> 
>> Look at a NetWare permission on some file and you can't tell what the
>> effective permission is, because it depends on inherited ones as well.
>> Look at an effective permission and you can't tell which parts of it
>> will change if you change the inherited permission.
> 
> MS solved that part by not allowing partially defined permissions.
> (At least AFAI can tell from the UI.)

The UI lies grossly, but in this area I think you are right. (I'm still
unclear as to what happens to permissions when you copy something:
do the inherited parts change, or not?)

>> Unix has inherited permissions in one sense: you can't get at a file
>> via some path if one of the directories in that path is unreadable.
> 
> NACK. ITYM non-accessable (-x), and it's only true if you can't reach it or
> one of the other links from $PWD and the directories you can fchdir to
> (which, admittedly, is the usural case).

Yeah, I was being sloppy. You got what I meant. (In similar wise, file
permissions don't stop you from reading a file if you can coerce someone
else into opening it and passing you its fd. Nonetheless, file
permissions almost always *do* work.)

>>>                                       File systems and individual
>>> directories should be able to be flagged as casesensitive/insensitive.
>> 
>> Only if you're willing to change POSIX to include a call to check filenames
>> for identity,
> 
> You'd "just" need a way to determine the canonialized form. Still an evil
> masterplan.

It'd still require changing POSIX and rewriting a large part of the
universe.

> [...]
>> It would also require case-conversion and locale-handling code, probably
>> including UTF-8 canoncalization code, inside the kernel. This would
>> greatly increase kernel complexity for a very small reward, and lead to
>> Al Viro's early death from cerebral aneurysm combined with involuntary
>> projectile vomiting. This cannot be considered a good thing.
> 
> a) NLS is in the kernel,

I don't think enough of it to do this is in there, at least not if you
aren't using something like NTFS.

>                          and if utf-8 filenames are supposed to be used,
>    an utf-8 checker rejecting non-canonialized strings will be desirable
>    to avoid binary trash in filenames or lookalike filenames. The
>    conversion to the canonialized form should happen outside the kernel.

Yes. But where? libc? (I can just see Ulrich going for *this*!)

> b) ACK, I don't think caseless handling of filenames is a good thing,
>    it would needlessly bloat the kernel by opening a can of worms.
>    E.g. 'ß' would be converted to 'SS'[0] in German or 'B' in greek.

... which means that either you lose per-process locale-dependence
via LANG et all, or you get the possibility of directories containing
several files with the same name from some users' POV.

Neither seems good to me; even though we already have part of this with
NTFS, we should not inflict it on people needlessly.

>> Now /etc/mtab, *that* is an abomination, and a small kernel improvement
>> (allowing arbitrary flag strings to be passed by mount into the kernel
>> solely for display in the appropriate /proc/mounts field) could
>> eliminate it and replace it entirely with /proc/mounts.
> 
> What about making all fs ignore the
> -omount="$tool:foo:bar;$tool2=baz:barf..." parameter?

Well, they'd have to dump it into /proc/mounts, too (I guess that would
happen magically). That would unbreak the few things this cares about.

> This is a cruel hack, but it will be backward compatible.
> (If your hammer is big enough, the problem may turn out to be a nail.)

Indeed. I'm fairly sure this problem is trivially solvable: it's far
easier than the *other* problems lying in wait down the per-user-
namespace path. :)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Bodo Eggert]

On Wed, 5 Oct 2005, Nix wrote:
> On Wed, 05 Oct 2005, Bodo Eggert suggested tentatively:

[...]
> >                          and if utf-8 filenames are supposed to be used,
> >    an utf-8 checker rejecting non-canonialized strings will be desirable
> >    to avoid binary trash in filenames or lookalike filenames. The
> >    conversion to the canonialized form should happen outside the kernel.
> 
> Yes. But where? libc? (I can just see Ulrich going for *this*!)

Either there or in a seperate library.

> > b) ACK, I don't think caseless handling of filenames is a good thing,
> >    it would needlessly bloat the kernel by opening a can of worms.
> >    E.g. 'ß' would be converted to 'SS'[0] in German or 'B' in greek.
> 
> ... which means that either you lose per-process locale-dependence
> via LANG et all, or you get the possibility of directories containing
> several files with the same name from some users' POV.
> 
> Neither seems good to me; even though we already have part of this with
> NTFS, we should not inflict it on people needlessly.

ACK, that's my point.

-- 
Field experience is something you don't get until just after you need it. 

Re: what's next for the linux kernel?

[Joe Bob Spamtest]

Marc Perkel wrote:
> Right - that's Unix "inside the box" thinking. The idea is to make the 
> operating system smarter so that the user doesn't have to deal with 
> what's computer friendly - but reather what makes sense to the user. 
>  From a user's perspective if you have not rights to access a file then 
> why should you be allowed to delete it?

then that file shouldn't be in a directory owned by $otheruser.

> Now - the idea is to create choice. If you need to emulate Unix nehavior 
> for compatibility that's fine. But I would migrate away from that into a 
> permissions paradygme that worked like Netware.

the word is 'paradigm.' anyhow, through posix acls and selinux you can
achieve the behaviour you so love.

> I started with Netware and I'm spoiled. They had it right 15 years ago 
> and Linux isn't any where near what I was with Netware and DOS in 1990. 
> Once you've had this kind of permission power Linux is a real big step 
> down.

if you like netware so much, then fucking use it. nobody here will stop you.

> So - the thread is about the future so I say - time to fix Unix.

UNIX isn't broken. you're just not asking it to do the right things.

Re: what's next for the linux kernel?

[Nikolay N. Ivanov]

Hi All!

> Time to fix Unix? I doubt something seriously borked would have 
> outlasted every other OS on the market. Unix was around before Netware 
> and, IMHO, will be around a long time after the last adherents of 
> NetWare are gone. (and with MS doing it's level best to kill NetWare 
> with it's own shared filesystems and built-in networking this cannot 
> be that far off. After all, Netware was developed to fill a vacancy in 
> the MS world.)
>
>  
>
Respect!

Nikolay N. Ivanov

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

okay, people: any chance of calming / ending / moving this rather
distracting thread?

davem - any chance of that LVML?

anyone know where dave works who is willing to kick him before
everyone goes nuts?

Re: what's next for the linux kernel?

[David S. Miller]

From: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date: Thu, 6 Oct 2005 01:03:40 +0100

> okay, people: any chance of calming / ending / moving this rather
> distracting thread?
> 
> davem - any chance of that LVML?
> 
> anyone know where dave works who is willing to kick him before
> everyone goes nuts?

I still don't see any good reason to make a new mailing list.

People argue here because they get a large audience.  If we
create a new mailing list, the vast majority of folks won't
take these threads there because they'll get less of the attention
they're after.

Why don't we just kill off this silly thread instead? :-)

Re: what's next for the linux kernel?

[Nigel Rantor]

David S. Miller wrote:
> From: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
> Date: Thu, 6 Oct 2005 01:03:40 +0100
> 
> 
>>okay, people: any chance of calming / ending / moving this rather
>>distracting thread?
>>
>>davem - any chance of that LVML?
>>
>>anyone know where dave works who is willing to kick him before
>>everyone goes nuts?
> 
> 
> I still don't see any good reason to make a new mailing list.
> 
> People argue here because they get a large audience.  If we
> create a new mailing list, the vast majority of folks won't
> take these threads there because they'll get less of the attention
> they're after.
> 
> Why don't we just kill off this silly thread instead? :-)

Hell no, I need something to read while I'm cogitating on work and 
waiting for things to compile...any chance we can get all the major 
participants into a room and televise it as a cage match?

*popcorn*

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 05:14:55PM -0700, David S. Miller wrote:
> From: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
> Date: Thu, 6 Oct 2005 01:03:40 +0100
> 
> > okay, people: any chance of calming / ending / moving this rather
> > distracting thread?
> > 
> > davem - any chance of that LVML?
> > 
> > anyone know where dave works who is willing to kick him before
> > everyone goes nuts?
> 
> I still don't see any good reason to make a new mailing list.
> 
> People argue here because they get a large audience.  If we
> create a new mailing list, the vast majority of folks won't
> take these threads there because they'll get less of the attention
> they're after.

 oh go on: i promise i'll even subscribe to it.  and lots of other
 people will, too [won't we, folks YEERRRRS there you go, dave, see?]

 we want the _right_ kind of attention, and _you_ need somewhere to
 kindly but firmly push the people who are a bit brighter than kernel
 newbies but who still talk 19-to-the-dozen without posting a patch.

 i wouldn't even mind if there was an auto-redirect of certain
 subject lines duly noted "as of right now, all messages with
 this subject are being auto-forwarded to LVML".


> Why don't we just kill off this silly thread instead? :-)

 nothing would give me more delight than to see that happen.


 pleaaase, dave: you could even set up a redirect of this subject line
 to forward the messages to LVML instead.


-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Al Viro]

On Thu, Oct 06, 2005 at 12:10:50PM +0100, Luke Kenneth Casson Leighton wrote:
>  oh go on: i promise i'll even subscribe to it.  and lots of other
>  people will, too [won't we, folks YEERRRRS there you go, dave, see?]

So why don't you run it on your own box?

>  we want the _right_ kind of attention, and _you_ need somewhere to
>  kindly but firmly push the people who are a bit brighter than kernel
>  newbies but who still talk 19-to-the-dozen without posting a patch.

Stop insulting kernel newbies.  There is a difference between "starting
to learn subject" and "picking buzzwords vaguely related to subject and
playing a Markov chain".  And the latter does _not_ mean being brighter,
no matter how much new-age bullshit you spin around it.

Re: what's next for the linux kernel?

[Howard Chu]

Howdy Luke...

Luke Kenneth Casson Leighton wrote:
> > If you can't do it with unix permissions or unix permissions + ACL,
> > you don't need to do it at all most likely, and even more likely
> > you

>  the bastion sftp example i gave which required selinux on top of a
>  much broader set of POSIX file permissions demonstrates the fallacy
>  of your statement.

>  try to achieve the same effect with POSIX - even POSIX ACLs (uploader
>  only has create and write, not read, not delete; downloader has read
>  and delete, not write, not create)

>  and you will fail, miserably, because under POSIX, write implies
>  create.

You're really muddying up the waters here. sftp is not POSIX. Like ftp, 
it presents an abstraction of a generic filesystem, and that abstraction 
lives at the application layer. As such, it is the application layer's 
responsibility to define what features may or may not be implemented. 
There are plenty of smart ftp servers that let you define precisely this 
type of ACLs for the ftp users. The fact that it is both possible and 
trivially easy to implement such an application on top of a POSIX 
runtime environment tells me that there's nothing broken here.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 05:37:59PM -0400, Chris wrote:

> You write:
> |  personally i find that i like a bit of a run-up and/or advance notice
> |  of major paradigm shifts.
> 
>  I maintain that one of the problems is that we simply do not know how
> to efficiently (in all senses of the word) program multi-processing
> systems and applications. Various people have various theories about how
> it should be done, but they are not well-proven.
> 
>  We do know that the current approach is difficult and error-prone, but
> terribly attractive because of an at least superficial simplicity. So
> far none of the alternatives have been convincing enough to supplant it.
> 
>  Efficient changing of paradigms requires a target paradigm. In this
> environment, a good target paradigm does not appear to exist yet and
> may not be clear for years. Starting engineering work seems a little
> bit premature in that situation.
 
 chris,

 your words paraphrase nicely the issues faced - for software engineers.

 however, if you were to speak to a hardware engineer - an embedded
 systems designer of ASICs - they would have a completely different
 take on it, because they are dealing with parallelism all the time,
 _and_ they even have some tools to help them do that.

 what those ASIC designers lack is an off-the-shelf affordable
 multiprocessor chip that makes their parallel algorithms run
 fast enough: they _have_ to go for custom ASIC.

 which is where the pricing of 90nm - $250k mask charges making up most
 of the $2m NREs - is so key.  even .13micron you're looking at $200k
 NREs [non-recoverable expenditure].
 
 and if you were to do a chip at .13micron you would be
 looking at 120 watts for something running at 800mhz with
 only 1 million gates.
 
 a pentium 3 800mhz in other words, and that won't exactly get you very
 far.


 what the software engineers you refer to above lack is the toolchain to
 assist them in developing for anything other than uniprocessor targets.

 such tools and techniques are being researched and developed (i sent
 references to a coupl and also a google search criteria in another
 earlier LKML email): however, yet again, it's chicken-and-egg.

 until the chips start going multiprocessor, the tools are going to
 remain in research labs.  until the tools come out of research labs,
 the chips are going to remain useless.

 where it all goes a bit pearshaped with that chicken-and-egg vicious
 cycle is if the bottom drops out of 65nm and 45nm processes, such that
 _even_ the top uniprocessor mass-market chip manufacturers are forced
 down a parallel processing line.

 my point is: we're starting to see evidence of that happening
 (small-scale, 2-cores, 2-hyperthreads, talk of 4-cores, etc.
  even the X-Box 360 PPC 3x2)

 _therefore_, i invite people who do linux kernel development
 to think ahead - to take a _lead_ for once instead of waiting
 for hardware to drop into their laps, at which point it is once
 again too late, the hardware design decisions will have
 already been made by someone else, and you will be treated
 like second class citizens.  again.

 l.

Re: what's next for the linux kernel?

[Jens Axboe]

On Wed, Oct 05 2005, Luke Kenneth Casson Leighton wrote:
>  _therefore_, i invite people who do linux kernel development
>  to think ahead - to take a _lead_ for once instead of waiting
>  for hardware to drop into their laps, at which point it is once
>  again too late, the hardware design decisions will have
>  already been made by someone else, and you will be treated
>  like second class citizens.  again.

You seem to be suffering the same fate as other 'pocket visionaries'
with grand visions on this mailing list. The combination of 'listen to
this awesome idea' (*) and someone who hasn't contributed anything
interesting is not very good. Add a pinch (or more) of 'I know better'
and you've got a great cocktail for a lengthy and tiresome thread, but
not for anything productive.

Why is that so hard to understand? Succesful contributions start at the
technical level, always have.

(*) Such an idea is often described in great text detail, but is often
greatly lacking in technical detail.

-- 
Jens Axboe

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 02:31:14PM +0200, Jens Axboe wrote:

> [i know better criticism and accusations deleted and not commented on]

> Why is that so hard to understand? Succesful contributions start at the
> technical level, always have.
 
 then we will have to agree to disagree, because i believe that
 successful contributions start with "what creative thing shall
 we do now / what problem shall we tackle today in a creative
 way?" and work their way down to the technical level, which,
 as you rightly point out, requires successful _technical_
 contributions.

 l.

Re: what's next for the linux kernel?

[Jens Axboe]

On Wed, Oct 05 2005, Luke Kenneth Casson Leighton wrote:
> On Wed, Oct 05, 2005 at 02:31:14PM +0200, Jens Axboe wrote:
> 
> > [i know better criticism and accusations deleted and not commented on]
> 
> > Why is that so hard to understand? Succesful contributions start at the
> > technical level, always have.
>  
>  then we will have to agree to disagree, because i believe that
>  successful contributions start with "what creative thing shall
>  we do now / what problem shall we tackle today in a creative
>  way?" and work their way down to the technical level, which,
>  as you rightly point out, requires successful _technical_
>  contributions.

It's not my opinion, I'm stating historical fact. I honestly can't
remember when a succesfull contribution was made based on a long
non-technical thread. I can, however, remember oodles of cases where the
reverse is quite true.

-- 
Jens Axboe

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 03:40:43PM +0200, Jens Axboe wrote:
> On Wed, Oct 05 2005, Luke Kenneth Casson Leighton wrote:
> > On Wed, Oct 05, 2005 at 02:31:14PM +0200, Jens Axboe wrote:
> > 
> > > [i know better criticism and accusations deleted and not commented on]
> > 
> > > Why is that so hard to understand? Succesful contributions start at the
> > > technical level, always have.
> >  
> >  then we will have to agree to disagree, because i believe that
> >  successful contributions start with "what creative thing shall
> >  we do now / what problem shall we tackle today in a creative
> >  way?" and work their way down to the technical level, which,
> >  as you rightly point out, requires successful _technical_
> >  contributions.
> 
> It's not my opinion, I'm stating historical fact. I honestly can't
> remember when a succesfull contribution was made based on a long
> non-technical thread. I can, however, remember oodles of cases where the
> reverse is quite true.
 
 ah.

 who's hosting the linux-visionaries mailing list, then?

Re: what's next for the linux kernel?

[Jens Axboe]

On Wed, Oct 05 2005, Luke Kenneth Casson Leighton wrote:
> On Wed, Oct 05, 2005 at 03:40:43PM +0200, Jens Axboe wrote:
> > On Wed, Oct 05 2005, Luke Kenneth Casson Leighton wrote:
> > > On Wed, Oct 05, 2005 at 02:31:14PM +0200, Jens Axboe wrote:
> > > 
> > > > [i know better criticism and accusations deleted and not commented on]
> > > 
> > > > Why is that so hard to understand? Succesful contributions start at the
> > > > technical level, always have.
> > >  
> > >  then we will have to agree to disagree, because i believe that
> > >  successful contributions start with "what creative thing shall
> > >  we do now / what problem shall we tackle today in a creative
> > >  way?" and work their way down to the technical level, which,
> > >  as you rightly point out, requires successful _technical_
> > >  contributions.
> > 
> > It's not my opinion, I'm stating historical fact. I honestly can't
> > remember when a succesfull contribution was made based on a long
> > non-technical thread. I can, however, remember oodles of cases where the
> > reverse is quite true.
>  
>  ah.
> 
>  who's hosting the linux-visionaries mailing list, then?

:-)

Don't think there is one, but if it will take the traffic of this list,
I'm sure davem can be quickly convinced to add one!

-- 
Jens Axboe

Re: what's next for the linux kernel?

[Steven Rostedt]

On Wed, 5 Oct 2005, Jens Axboe wrote:

> >
> >  who's hosting the linux-visionaries mailing list, then?
>
> :-)
>
> Don't think there is one, but if it will take the traffic of this list,
> I'm sure davem can be quickly convinced to add one!
>

Amen!

(noise to content is very high lately)

And the sad part is that I keep reading this crap.  It's like being sick
at home one day and getting stuck on a bad soap opera.  You know the stuff
you're watching is garbage, but you just can't get away and stop watching
it.

At least if it moved to another list I'd be strong enough not to
subscribe. :-)

-- Steve

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 12:38:35PM -0400, Steven Rostedt wrote:
> At least if it moved to another list I'd be strong enough not to
> subscribe. :-)
 
 and i'd feel less guilty about replying oh crap.

Re: what's next for the linux kernel?

[Dave Neuer]

On 10/5/05, Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
>
>  where it all goes a bit pearshaped with that chicken-and-egg vicious
>  cycle is if the bottom drops out of 65nm and 45nm processes, such that
>  _even_ the top uniprocessor mass-market chip manufacturers are forced
>  down a parallel processing line.
>
>  my point is: we're starting to see evidence of that happening
>  (small-scale, 2-cores, 2-hyperthreads, talk of 4-cores, etc.
>  even the X-Box 360 PPC 3x2)
>
>  _therefore_, i invite people who do linux kernel development
>  to think ahead - to take a _lead_ for once instead of waiting
>  for hardware to drop into their laps, at which point it is once
>  again too late, the hardware design decisions will have
>  already been made by someone else, and you will be treated
>  like second class citizens.  again.

With all due respect (and I do believe some is due); your comment
above makes no sense. Operating system designers design software to
operate on existing systems -- often doing as much as possible to
ensure that the design supports lots of different systems. However, _a
neccessary prerequisite_ for that activity is "hardware dropping into
their laps." You warn that "the hardware design decisions will have
already been made by someone else;" but that is the order in which it
_neccessarily_ works! What hardware designer out there spends his day
looking around for as-yet-unused (and hence untested) operating
systems that run on some hypothetical yet-to-be-designed hardware and
say to themselves, "ah, that looks nice -- I think I can design some
hardware that will run that software _really well_?"

Your argument defies logic, and it's that fact -- not your intentions
-- that makes this thread so tiresome.

Dave

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

>  With all due respect (and I do believe some is due); your comment above
> makes no sense. 

 replying privately. l.

RE: what's next for the linux kernel?

[Martin Fouts]

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Luke 
> Kenneth Casson Leighton
> Sent: Monday, October 03, 2005 1:31 PM
> To: Joe Bob Spamtest
> Cc: linux-kernel
> Subject: Re: what's next for the linux kernel?
> 
> On Mon, Oct 03, 2005 at 12:36:20PM -0700, Joe Bob Spamtest wrote:
> > The point being: If and when the industry switches its focus to 
> > highly parallel systems, Linux will shortly follow.
> 
> 
> personally i find that i like a bit of a run-up and/or 
> advance notice  of major paradigm shifts.  on the basis that 
> other people might also  want to know, i initiated this 
> discussion yesterday and it seems like  forever already! :)

There's no real need to hurry.  The industry isn't going to shift its
focus to highly parallel systems in the near future.  Highly parallel
systems have been around since the 60s, (when 'highly parallel' meant a
lot less than it does today, but was still parallel compared to typical
von neumann machines of the period.)

It's been fifteen years since I last played with them, and there's not
much chance that they'll get interesting in the near future.

They'll never go completely away, because there'll always be niches
where they make sense, but they'll never break out into the mainstream,
because those niches tend to get smaller, not larger, over time.

what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

hi,

as i love to put my oar in where it's unlikely that people
will listen, and as i have little to gain or lose by doing
so, i figured you can decide for yourselves whether to be
selectively deaf or not:

here's my take on where i believe the linux kernel needs to go,
in order to keep up.

what prompted me to send this message now was a recent report
where linus' no1 patcher is believed to be close to overload,
and in that report, i think it was andrew morton, it was
said that he believed the linux kernel development rate to be
slowing down, because it is nearing completion.

i think it safe to say that a project only nears completion
when it fulfils its requirements and, given that i believe that
there is going to be a critical shift in the requirements, it
logically follows that the linux kernel should not be believed
to be nearing completion.

with me so far? :)

okay, so what's the bit that's missing that mr really irritating,
oh-so-right and oh-so-killfile-ignorable luke kenneth casson
kipper lozenge has spotted that nobody else has, and what's
the fuss about?

well... to answer that, i need to outline a bit about processor
manufacturing: if you are familiar with processor design please
forgive me, this is for the benefit of those who might not be.

the basic premise: 90 nanometres is basically... well...
price/performance-wise, it's hit a brick wall at about 2.5Ghz, and
both intel and amd know it: they just haven't told anyone.

anyone (big) else has a _really_ hard time getting above 2Ghz,
because the amount of pipelining required is just... insane
(see recent ibm powerpc5 see slashdot - what speed does it do?
surprise: 2.1Ghz when everyone was hoping it would be 2.4-2.5ghz).

a _small_ chip design company (not an IBM, intel or amd)
will be lucky to see this side of 1Ghz, at 90nm.

also, the cost of mask charges for 90nm is insane: somewhere
around $2million and that's never going to go away.

the costs for 65nm are going to be far far greater than that,
and 45nm i don't even want to imagine what they're going to be.

plus, there's a problem of quantum mechanics, heat dissipation
and current drain that makes, with current manufacturing
techniques, the production of 65nm and 45nm chips really
problematic.

with present manufacturing techniques, the current drain and heat
dissipation associated with 45nm means that you have to cut the number
of gates down to ONE MILLION, otherwise the chip destroys itself.

(brighter readers might now have an inkling of where i'm going
with this - bear with me :)

compare that one million gates with the present number of gates in an
AMD or x86 chip - some oh, what, 20 million?

now you get it?

for the present insane uniprocessor architectures at least
(and certainly for the x86 design), 90nm is _it_ - and yet,
people demand ever more faster and faster amounts of processing,
and no amount of trying on the part of engineers can get round
the laws of physics.

so, what's the solution?

well.... it's to back to parallel processing techniques, of course.

and, surprise surprise, what do we have intel pushing?

apart from, of course, the performance per watt metric (which,
if you read a few paragraphs back, you realise why they have to
trick both their customers and their engineers into believing
that performance/watt is suddenly important, it's because they
have to carve a path for a while getting the current usage down
in order for the 65nm chips to become palatable - assuming they
can be made at all in a realistic yield - read price bracket)

well - intel is pushing "hyperthreading".

and surprise, surprise, what is amd pushing?  dual-core chips.

and what is in the X-Box 360?  a PowerPC _triple_ core, _dual_
hyper-threaded processor!!

i believe that the X-Box 360 processor is the way things
are going to be moving - quad-core quad-threaded processors;
16 and 32 core ultra-RISC processors: medium to massive parallel
processors, but this time single-chip unlike the past decade(s) where
multi-core was hip and cool and... expensive.

i believe the future to contain stacks of single-chip multiprocessing
designs in several forms - including intel's fun-and-games VLIW stuff.

remember: intel recently bought the company that has spent
15 years working on that DEC/Alpha just-in-time x86-to-alpha
assembly converter product (remember DEC/Alphas running NT 3.51,
anyone, and still being able to run x86 programs?)

and, what is the linux kernel?

it's a daft, monolithic design that is suitable and faster on
single-processor systems, and that design is going to look _really_
outdated, really soon.

fortunately, there is a research project that has already
done a significant amount of work in breaking away from the
monolithic design: the l4linux project.

last time i checked, a few months ago, they were keeping thoroughly
up-to-date and had either 2.6.11 or 2.6.12 ported, can't recall which.

the l4linux project puts the linux kernel on top of L4-compliant
microkernels (yes, just like the gnu hurd :) and there are several such
L4-compliant microkernels - named after nuts.  pistachio, etc.

one of those l4-compliant microkernels is a parallel processor
based one - it's SMP compliant, it even has support for virtual
machining, whoopee, ain't that fun.

i remember now.  university of south australia, and university
of karlsruhe.  i probably spelled that wrong.


in short, basically, if you follow and agree with the logic, the
linux kernel - as maintained by linus - is far from complete.

i therefore invite you to consider the following strategy:

1) that the linux kernel should merge with the oskit project or that the
linux kernel should split into two projects - a) 30-40k lines of code
comprising the code in kernel/* and headers and ports and headers
b) device drivers i.e duh the oskit project.

2) that the linux kernel should merge and maintain the efforts
of the l4linux project as mainlined not sidelined.

3) that serious efforts be diverted into the l4 microkernels to make it
portable, work on parallel processor systems, hyperthreaded, SMP and
other (such as ACPI which has had to be #ifdef'd out even in XEN).

4) other.

yes, i know this flies in the face of linus' distaste for
message-based kernels, and it's because message-passing slows
things down... but they slow things down _only_ on uniprocessor
kernel designs, and uniprocessors are going to be blowing
goats / bubbles / insert-as-appropriate in the not-too-distant
future.  there have _already_ been high-profile parallel
processor designs announced, released, and put into service
(e.g. dual-core AMD64, triple-core dual-hyperthreaded PowerPC in
the X-Box 360).

yes, i may have got things wrong.

yes, it is up to _you_ to point them out.

yes, it is up to _you_ to decide what to do, not me.

good luck.

l.

p.s. XEN is even getting lovely encouraging noises from intel
to support hyperthreading, isn't that nice boys and girls?

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Rik van Riel]

On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:

> and, what is the linux kernel?
> 
> it's a daft, monolithic design that is suitable and faster on
> single-processor systems, and that design is going to look _really_
> outdated, really soon.

Linux already has a number of scalable SMP synchronisation
mechanisms. The main scalability effort nowadays is about
the avoidance of so-called "cache line bouncing".

http://wiki.kernelnewbies.org/wiki/SMPSynchronisation

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
> On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:
> 
> > and, what is the linux kernel?
> > 
> > it's a daft, monolithic design that is suitable and faster on
> > single-processor systems, and that design is going to look _really_
> > outdated, really soon.
> 
> Linux already has a number of scalable SMP synchronisation
> mechanisms. 

 ... and you are tied in to the decisions made by the linux kernel
 developers.

 whereas, if you allow something like a message-passing design (such as
 in the port of the linux kernel to l4), you have the option to try out
 different underlying structures - _without_ having to totally redesign
 the infrastructure.

 several people involved with the l4linux project have already done
 that: as i mentioned in the original post, there are about three or
 four different and separate l4 microkernels available for download
 (GPL) and one of them is ported to stacks of different architectures,
 and one of them is SMP capable and even includes a virtual machine
 environment.
 
 and they're only approx 30-40,000 lines each, btw.


 also, what about architectures that have features over-and-above SMP?

 in the original design of SMP it was assumed that if you have
 N processors that you have N-way access to memory.

 what if, therefore, someone comes up with an architecture that is
 better than or improves greatly upon SMP?

 they will need to make _significant_ inroads into the linux kernel
 code, whereas if, say, you oh i dunno provide hardware-accelerated
 parallel support for a nanokernel (such as l4) which just _happens_
 to be better than SMP then running anything which is l4 compliant gets
 the benefit.


 the reason i mention this is because arguments about saying "SMP is it,
 SMP is great, SMP is everything, we're improving our SMP design" don't
 entirely cut it, because SMP has limitations that don't scale properly
 to say 64 or 128 processors: sooner or later someone's going to come up
 with something better than SMP and all the efforts focussed on making
 SMP better in the linux kernel are going to look lame.

 l.

 p.s. yes i do know of a company that has improved on SMP.

Re: what's next for the linux kernel?

[Rik van Riel]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:

> > Linux already has a number of scalable SMP synchronisation
> > mechanisms. 
> 
>  ... and you are tied in to the decisions made by the linux kernel
>  developers.
> 
>  whereas, if you allow something like a message-passing design (such as
>  in the port of the linux kernel to l4), you have the option to try out
>  different underlying structures - _without_ having to totally redesign
>  the infrastructure.

Infrastructure is not what matters when it comes to SMP
scalability on modern systems, since lock contention is
not the primary SMP scalability problem.

Due to the large latency ratio between L1/L2 cache and
RAM, the biggest scalability problem is cache invalidation
and cache bounces.

Those are not solvable by using another underlying
infrastructure - they require a reorganization of the
datastructures on top, the data structures in Linux.

Note that message passing is by definition less efficient
than SMP synchronisation mechanisms that do not require
data to be exchanged between CPUs, eg. RCU or the use of
cpu-local data structures.

>  p.s. yes i do know of a company that has improved on SMP.

SGI ?  IBM ?

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 07:26:21PM -0400, Rik van Riel wrote:
> On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> > On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
> 
> > > Linux already has a number of scalable SMP synchronisation
> > > mechanisms. 
> > 
> >  ... and you are tied in to the decisions made by the linux kernel
> >  developers.
> > 
> >  whereas, if you allow something like a message-passing design (such as
> >  in the port of the linux kernel to l4), you have the option to try out
> >  different underlying structures - _without_ having to totally redesign
> >  the infrastructure.
> 
> Infrastructure is not what matters when it comes to SMP
> scalability on modern systems, since lock contention is
> not the primary SMP scalability problem.
> 
> Due to the large latency ratio between L1/L2 cache and
> RAM, the biggest scalability problem is cache invalidation
> and cache bounces.
> 
> Those are not solvable by using another underlying
> infrastructure - they require a reorganization of the
> datastructures on top, the data structures in Linux.

 ... ah, but what about in hardware?  what if you had hardware support
 for

 _plus_ what if you had some other OS primitives implemented
 in hardware, the use of which allowed you to avoid or minimise
 cache invalidation problems?

 not entirely, of course, but enough to make up for SMP's deficiencies.


> Note that message passing is by definition less efficient
> than SMP synchronisation mechanisms that do not require
> data to be exchanged between CPUs, eg. RCU or the use of
> cpu-local data structures.

 how about message passing by reference - a la c++? 

 i.e. using an "out-of-band" parallel message bus, you pass
 the address in a NUMA or SMP area of memory that is granted
 to a specific processor, which says to another processor oh
 something like "you now have access to this memory: by the time
 you get this message i will have already cleared the cache so
 you can get it immediately".

 that sort of thing.

 _and_ you use the parallel message bus to communicate memory
 allocation, locking, etc.

 _and_ you use the parallel message bus to implement semaphores and
 mutexes.

 _and_ if the message is small enough, you just pass the message across
 without going via external memory.

 
 ... but i digress - but enough to demonstrate, i hope, that
 this isn't some "pie-in-the-sky" thing, it's one hint at a
 solution to the problem that a lot of hardware designers haven't
 been able to solve, and up until now they haven't had to even
 _consider_ it.

 and they've avoided the problem by going "multi-core" and going
 "hyperthreading".

 but, at some point, hyperthreading isn't going to cut it, and at some
 point multi-core isn't going to cut it.

 and people are _still_ going to expect to see the monster
 parallelism (32, 64, 128 parallel hardware threads) as
 "one processor".

 the question is - and i iterate it again: can the present
 linux kernel design _cope_ with such monster parallelism?

 answer, at present, as maintained as-it-is, not a chance.

 question _that_ raises: do you _want_ to [make it cope with such
 monster parallelism]?

 and if the answer to that is "no, definitely not", then the
 responsibility can be offloaded onto a microkernel, e.g. the L4
 microkernel, and it _just_ so happens that the linux kernel has already
 been ported to L4.

 i raise this _one_ route - there are surely going to be others.

 i invite you to consider discussing them.
 
 LIKE FRIGGIN ADULTS, unlike the very spiteful comments i've
 received indicate that some people would like to do (no i
 don't count you in that number, rik, just in case you thought
 i was because i'm replying direct to you!).


> >  p.s. yes i do know of a company that has improved on SMP.
> 
> SGI ?  IBM ?
 
 no, they're a startup.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Rik van Riel]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:

>  how about message passing by reference - a la c++? 

>  _and_ you use the parallel message bus to communicate memory
>  allocation, locking, etc.

Then you lose.  It's the act of passing itself that causes
scalability problems and a loss of performance.

The best way to get SMP scalability is to avoid message
passing alltogether, using things like per-cpu data
structures and RCU.

Not having to pass a message is faster than any message
passing mechanism.

>  ... but i digress - but enough to demonstrate, i hope, that
>  this isn't some "pie-in-the-sky" thing,

You've made a lot of wild claims so far, most of which I'm
not ready to belief without some proof to back them up.

>  it's one hint at a solution to the problem that a lot of hardware
>  designers haven't been able to solve, and up until now they haven't
>  had to even _consider_ it.

The main problem is that communication with bits of silicon
four inches away is a lot slower, or takes much more power,
than communication with bits of silicon half a millimeter away.

This makes cross-core communication, and even cross-thread
communication in SMT/HT, slower than not having to have such
communication at all.

>  the question is - and i iterate it again: can the present
>  linux kernel design _cope_ with such monster parallelism?

The SGI and IBM people seem fairly happy with current 128 CPU
performance, and appear to be making serious progress towards
512 CPUs and more.

>  question _that_ raises: do you _want_ to [make it cope with such
>  monster parallelism]?
> 
>  and if the answer to that is "no, definitely not", then the
>  responsibility can be offloaded onto a microkernel,

No, that cannot be done, for all the reasons I mentioned
earlier in the thread.

Think about something like the directory entry cache (dcache),
all the CPUs need to see that cache consistently, and you cannot
avoid locking overhead by having the locking done by a microkernel.

The only way to avoid locking overhead is by changing the data
structure to something that doesn't need locking.

No matter how low your locking overhead - once you have 1024
CPUs it's probably too high.

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Vadim Lobanov]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:

> On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
> > On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:
> >
> > > and, what is the linux kernel?
> > >
> > > it's a daft, monolithic design that is suitable and faster on
> > > single-processor systems, and that design is going to look _really_
> > > outdated, really soon.
> >
> > Linux already has a number of scalable SMP synchronisation
> > mechanisms.
>
>  ... and you are tied in to the decisions made by the linux kernel
>  developers.
>
>  whereas, if you allow something like a message-passing design (such as
>  in the port of the linux kernel to l4), you have the option to try out
>  different underlying structures - _without_ having to totally redesign
>  the infrastructure.
>
>  several people involved with the l4linux project have already done
>  that: as i mentioned in the original post, there are about three or
>  four different and separate l4 microkernels available for download
>  (GPL) and one of them is ported to stacks of different architectures,
>  and one of them is SMP capable and even includes a virtual machine
>  environment.
>
>  and they're only approx 30-40,000 lines each, btw.
>
>
>  also, what about architectures that have features over-and-above SMP?
>
>  in the original design of SMP it was assumed that if you have
>  N processors that you have N-way access to memory.
>
>  what if, therefore, someone comes up with an architecture that is
>  better than or improves greatly upon SMP?

Like NUMA?

>  they will need to make _significant_ inroads into the linux kernel
>  code, whereas if, say, you oh i dunno provide hardware-accelerated
>  parallel support for a nanokernel (such as l4) which just _happens_
>  to be better than SMP then running anything which is l4 compliant gets
>  the benefit.
>
>
>  the reason i mention this is because arguments about saying "SMP is it,
>  SMP is great, SMP is everything, we're improving our SMP design" don't
>  entirely cut it, because SMP has limitations that don't scale properly
>  to say 64 or 128 processors: sooner or later someone's going to come up
>  with something better than SMP and all the efforts focussed on making
>  SMP better in the linux kernel are going to look lame.
>
>  l.
>
>  p.s. yes i do know of a company that has improved on SMP.
>
> -

-Vadim Lobanov

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:

> >  what if, therefore, someone comes up with an architecture that is
> >  better than or improves greatly upon SMP?
> 
> Like NUMA?
 
 yes, like numa, and there is more.

 i had the honour to work with someone who came up with a radical
 enhancement even to _that_.

 basically the company has implemented, in hardware (a
 nanokernel), some operating system primitives, such as message
 passing (based on a derivative by thompson of the "alice"
 project from plessey, imperial and manchester university
 in the mid-80s), hardware cache line lookups (which means
 instead of linked list searching, the hardware does it for
 you in a single cycle), stuff like that.

 the message passing system is designed as a parallel message bus -
 completely separate from the SMP and NUMA memory architecture, and as
 such it is perfect for use in microkernel OSes.

 (these sorts of things are unlikely to make it into the linux kernel, no
 matter how much persuasion and how many patches they would write).

 _however_, a much _better_ target would be to create an L4 microkernel
 on top of their hardware kernel.

 this company's hardware is kinda a bit difficult for most people to get
 their heads round: it's basically parallelised hardware-acceleration for
 operating systems, and very few people see the point in that.

 however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
 and to get any faster you _have_ to go parallel.

 and the drive for "faster", "better", "more sales" means more and more
 parallelism.

 it's _happening_ - and SMP ain't gonna cut it (which is why
 these multi-core chips are coming out and why hyperthreading
 is coming out).

 so.

 this is a heads-up.

 what you choose to do with this analysis is up to you.

 l.

Re: what's next for the linux kernel?

[Vadim Lobanov]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:

> On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:
>
> > >  what if, therefore, someone comes up with an architecture that is
> > >  better than or improves greatly upon SMP?
> >
> > Like NUMA?
>
>  yes, like numa, and there is more.

The beauty of capitalization is that it makes it easier for others to
read what you have to say.

>  i had the honour to work with someone who came up with a radical
>  enhancement even to _that_.
>
>  basically the company has implemented, in hardware (a
>  nanokernel), some operating system primitives, such as message
>  passing (based on a derivative by thompson of the "alice"
>  project from plessey, imperial and manchester university
>  in the mid-80s), hardware cache line lookups (which means
>  instead of linked list searching, the hardware does it for
>  you in a single cycle), stuff like that.

That sounds awesome, but I have something better -- a quantum computer.
And it's about as parallel as you're going to get anytime in the
foreseeable future!

...

Moral of the story: There are thousands of hardware doodads all around.
People only start to become interested when they have actual "metal"
freely available on the market, that they can play with and code to.

>  the message passing system is designed as a parallel message bus -
>  completely separate from the SMP and NUMA memory architecture, and as
>  such it is perfect for use in microkernel OSes.

You're making an implicit assumption here that it will benefit _only_
microkernel designs. That is not at all immediate or obvious to me (or,
I suspect, others also) -- where's the proof?

>  (these sorts of things are unlikely to make it into the linux kernel, no
>  matter how much persuasion and how many patches they would write).

No, the kernel hackers are actually very sensible people. When they push
back, there's usually a darn good reason for it. See above point
regarding availability of hardware.

>  _however_, a much _better_ target would be to create an L4 microkernel
>  on top of their hardware kernel.

Perfect. You can do that, and benefit from the oodles of fame that
follow. Others might be less-than-convinced.

>  this company's hardware is kinda a bit difficult for most people to get
>  their heads round: it's basically parallelised hardware-acceleration for
>  operating systems, and very few people see the point in that.

That just sounds condescending.

>  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
>  and to get any faster you _have_ to go parallel.

Sure, it's going to stop somewhere, but you have to be a heck of a
visionary to predict that it will stop _there_. People have been
surprised before on such matters, so don't go around yelling about the
impending doom quite yet.

>  and the drive for "faster", "better", "more sales" means more and more
>  parallelism.
>
>  it's _happening_ - and SMP ain't gonna cut it (which is why
>  these multi-core chips are coming out and why hyperthreading
>  is coming out).

"Rah, rah, parallelism is great!" -- That's a great slogan, except...

Users, who also happen to be the target of those sales, care about
_userland_ applications. And the bitter truth is that the _vast_
majority of userland apps are single-threaded. Why? Two reasons --
first, it's harder to write a multithreaded application, and second,
some workloads simply can't be expressed "in parallel". Your kernel
might (might, not will) run like a speed-demon, but the userland stuff
will still be lackluster in comparison.

And that's when your slogan hits a wall, and the marketing hype dies.
The reality is that parallelism is something to be desired, but is not
always achievable.

>  so.
>
>  this is a heads-up.
>
>  what you choose to do with this analysis is up to you.

I choose to wait for actual, concrete details and proofs of your design,
instead of the ambiguous "visionary" hand-waving so far. As has already
been said, -ENOPATCH.

>  l.
>

-Vadim Lobanov

Re: what's next for the linux kernel?

[Al Viro]

On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:
> I choose to wait for actual, concrete details and proofs of your design,
> instead of the ambiguous "visionary" hand-waving so far. As has already
> been said, -ENOPATCH.

Speaking of which, IIRC, somebody used to maintain a list of words, acronyms,
etc. useful to know if you want to read l-k.  May I submit an addition to
that list?

visionary [n]: onanist with strong exhibitionist tendencies; from
"visions", the source of inspiration they refer to when it becomes
obvious that they have lost both sight and capacity for rational
thought.

Re: what's next for the linux kernel?

[Vadim Lobanov]

On Mon, 3 Oct 2005, Al Viro wrote:

> On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:
> > I choose to wait for actual, concrete details and proofs of your design,
> > instead of the ambiguous "visionary" hand-waving so far. As has already
> > been said, -ENOPATCH.
>
> Speaking of which, IIRC, somebody used to maintain a list of words, acronyms,
> etc. useful to know if you want to read l-k.  May I submit an addition to
> that list?
>
> visionary [n]: onanist with strong exhibitionist tendencies; from
> "visions", the source of inspiration they refer to when it becomes
> obvious that they have lost both sight and capacity for rational
> thought.
>

Nice. :-)

Just from idle curiosity, you wouldn't know where that list currently
resides, would you?

-Vadim Lobanov

Re: what's next for the linux kernel?

[Al Viro]

On Sun, Oct 02, 2005 at 06:50:46PM -0700, Vadim Lobanov wrote:
> > visionary [n]: onanist with strong exhibitionist tendencies; from
> > "visions", the source of inspiration they refer to when it becomes
> > obvious that they have lost both sight and capacity for rational
> > thought.
> >
> 
> Nice. :-)
> 
> Just from idle curiosity, you wouldn't know where that list currently
> resides, would you?

No idea...  Probably somebody from kernelnewbies.org crowd would know
the current location...

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 02:53:00AM +0100, Al Viro wrote:
> On Sun, Oct 02, 2005 at 06:50:46PM -0700, Vadim Lobanov wrote:
> > > visionary [n]: onanist with strong exhibitionist tendencies; from
> > > "visions", the source of inspiration they refer to when it becomes
> > > obvious that they have lost both sight and capacity for rational
> > > thought.
 
 oo, nice pretty flowers, wheeee :)

Re: what's next for the linux kernel?

[Erik Mouw]

On Mon, Oct 03, 2005 at 02:53:00AM +0100, Al Viro wrote:
> On Sun, Oct 02, 2005 at 06:50:46PM -0700, Vadim Lobanov wrote:
> > > visionary [n]: onanist with strong exhibitionist tendencies; from
> > > "visions", the source of inspiration they refer to when it becomes
> > > obvious that they have lost both sight and capacity for rational
> > > thought.
> > >
> > 
> > Nice. :-)
> > 
> > Just from idle curiosity, you wouldn't know where that list currently
> > resides, would you?
> 
> No idea...  Probably somebody from kernelnewbies.org crowd would know
> the current location...

There's not really a list of words on kernelnewbies.org, but a fortunes
file:

  http://www.kernelnewbies.org/kernelnewbies-fortunes.tar.gz

(and I just updated it with your description of a visionary)


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Eigen lab: Delftechpark 26, 2628 XH, Delft, Nederland
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:

> On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> 
> > On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:
> >
> > > >  what if, therefore, someone comes up with an architecture that is
> > > >  better than or improves greatly upon SMP?
> > >
> > > Like NUMA?
> >
> >  yes, like numa, and there is more.
> 
> The beauty of capitalization is that it makes it easier for others to
> read what you have to say.
 
 sorry, vadim: haven't touched a shift key in over 20 years.

> >  basically the company has implemented, in hardware (a
> >  nanokernel), some operating system primitives, such as message
> >  passing (based on a derivative by thompson of the "alice"
> >  project from plessey, imperial and manchester university
> >  in the mid-80s), hardware cache line lookups (which means
> >  instead of linked list searching, the hardware does it for
> >  you in a single cycle), stuff like that.
> 
> That sounds awesome, but I have something better -- a quantum computer.
> And it's about as parallel as you're going to get anytime in the
> foreseeable future!

 :)

 *sigh* - i _so_ hope we don't need degrees in physics to program
 them...

> >  the message passing system is designed as a parallel message bus -
> >  completely separate from the SMP and NUMA memory architecture, and as
> >  such it is perfect for use in microkernel OSes.
> 
> You're making an implicit assumption here that it will benefit _only_
> microkernel designs. 

 ah, i'm not: i just left out mentioning it :)

 the message passing needs to be communicated down to manage
 threads, and also to provide a means to manage semaphores and
 mutexes: ultimately, support for such an architecture would
 work its way down to libc.


 and yes, if you _really_ didn't want a kernel in the way at all, you
 could go embedded and just... do everything yourself.

 or port reactos, the free software reimplementation of nt,
 to it, or something :)

 *shrug*.

> >  this company's hardware is kinda a bit difficult for most people to get
> >  their heads round: it's basically parallelised hardware-acceleration for
> >  operating systems, and very few people see the point in that.
> 
> That just sounds condescending.
 
 i'm very sorry about that, it wasn't deliberate and ... re-reading
 my comment, i should say that my comment isn't actually entirely true!

 a correction/qualification: the people whom the startup company
 contacted before they were put in touch with me had found that
 everybody they had previously talked to just simply _did_ not
 get it: this was presumably because of their choice of people
 whom they were seeking funding from were not technically up
 to the job of understanding the concept.

 i didn't mean to imply that _everyone_ - or more specifically the
 people reading this list - would not get it.

 sorry.

> >  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
> >  and to get any faster you _have_ to go parallel.
> 
> Sure, it's going to stop somewhere, but you have to be a heck of a
> visionary to predict that it will stop _there_. 

 okay, i admit it: you caught me out - i'm a mad visionary.

 but seriously.

 it won't stop - but the price of 90nm mask charges, at approx
 $2m, is already far too high, and the number of large chips
 being designed is plummetting like a stone as a result - from
 like 15,000 per year a few years ago down to ... damn, can't remember -
 less than a hundred (i think!  don't quote me on that!)

 when 90 nm was introduced, some mad fabs wanted to make 9
 metre lenses, dude!!! until carl zeiss were called in and
 managed to get it down to 3 metres.

 and that lens is produced on a PER CHIP basis.

 basically, it's about cost.

 the costs of producing faster and faster uniprocessors is
 getting out of control.

 i'm not explaining things very well, but i'm trying.  too many words,
 not concise enough, too much to explain without people misunderstanding
 or skipping things and getting the wrong end of the stick.

 argh.


> >  and the drive for "faster", "better", "more sales" means more and more
> >  parallelism.
> >
> >  it's _happening_ - and SMP ain't gonna cut it (which is why
> >  these multi-core chips are coming out and why hyperthreading
> >  is coming out).
> 
> "Rah, rah, parallelism is great!" -- That's a great slogan, except...
> 
> Users, who also happen to be the target of those sales, care about
> _userland_ applications. And the bitter truth is that the _vast_
> majority of userland apps are single-threaded. Why? Two reasons --
> first, it's harder to write a multithreaded application, and second,
> some workloads simply can't be expressed "in parallel". Your kernel
> might (might, not will) run like a speed-demon, but the userland stuff
> will still be lackluster in comparison.
> 
> And that's when your slogan hits a wall, and the marketing hype dies.
> The reality is that parallelism is something to be desired, but is not
> always achievable.

 okay: i will catch up on this bit, another time, because it is late
 enough for me to be getting dizzy and appearing to be drunk.

 this is one answer (and there are others i will write another time.
 hint: automated code analysis tools, auto-parallelising tools, both
 offline and realtime):

 watch what intel and amd do: they will support _anything_ - clutch at
 straws - to make parallelism palable, why?  because in order to be
 competitive - and realistically priced - they don't have any choice.

 plus, i am expecting the chips to be thrown out there (like
 the X-Box 360 which has SIX hardware threads remember) and
 the software people to quite literally _have_ to deal with it.

 i expect the hardware people to go: this is the limit, this is what we
 can do, realistically price-performance-wise: lump it, deal with it.

 when intel and amd start doing that, everyone _will_ lump it.
 and deal with it.

 ... why do you think intel is hyping support for and backing
 hyperthreads support in XEN/Linux so much?

 l.

Re: what's next for the linux kernel?

[Vadim Lobanov]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:

> On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:
>
> > On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> >
> > > On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:
> > >
> > > > >  what if, therefore, someone comes up with an architecture that is
> > > > >  better than or improves greatly upon SMP?
> > > >
> > > > Like NUMA?
> > >
> > >  yes, like numa, and there is more.
> >
> > The beauty of capitalization is that it makes it easier for others to
> > read what you have to say.
>
>  sorry, vadim: haven't touched a shift key in over 20 years.

It's not going to bite you. I promise.

> > >  basically the company has implemented, in hardware (a
> > >  nanokernel), some operating system primitives, such as message
> > >  passing (based on a derivative by thompson of the "alice"
> > >  project from plessey, imperial and manchester university
> > >  in the mid-80s), hardware cache line lookups (which means
> > >  instead of linked list searching, the hardware does it for
> > >  you in a single cycle), stuff like that.
> >
> > That sounds awesome, but I have something better -- a quantum computer.
> > And it's about as parallel as you're going to get anytime in the
> > foreseeable future!
>
>  :)
>
>  *sigh* - i _so_ hope we don't need degrees in physics to program
>  them...
>
> > >  the message passing system is designed as a parallel message bus -
> > >  completely separate from the SMP and NUMA memory architecture, and as
> > >  such it is perfect for use in microkernel OSes.
> >
> > You're making an implicit assumption here that it will benefit _only_
> > microkernel designs.
>
>  ah, i'm not: i just left out mentioning it :)
>
>  the message passing needs to be communicated down to manage
>  threads, and also to provide a means to manage semaphores and
>  mutexes: ultimately, support for such an architecture would
>  work its way down to libc.
>
>
>  and yes, if you _really_ didn't want a kernel in the way at all, you
>  could go embedded and just... do everything yourself.
>
>  or port reactos, the free software reimplementation of nt,
>  to it, or something :)
>
>  *shrug*.

No, for reliability and performance reasons, I very much want a kernel
in the way. After all, kernel code is orders of magnitude better tuned
than almost all userland code.

The point I was making here is that, from what I can see, the current
Linux architecture is quite alright in anticipation of the hardware that
you're describing. It _could_ be better tuned for such hardware, sure,
but so far there is no need for such work at this particular moment.

> > >  this company's hardware is kinda a bit difficult for most people to get
> > >  their heads round: it's basically parallelised hardware-acceleration for
> > >  operating systems, and very few people see the point in that.
> >
> > That just sounds condescending.
>
>  i'm very sorry about that, it wasn't deliberate and ... re-reading
>  my comment, i should say that my comment isn't actually entirely true!
>
>  a correction/qualification: the people whom the startup company
>  contacted before they were put in touch with me had found that
>  everybody they had previously talked to just simply _did_ not
>  get it: this was presumably because of their choice of people
>  whom they were seeking funding from were not technically up
>  to the job of understanding the concept.
>
>  i didn't mean to imply that _everyone_ - or more specifically the
>  people reading this list - would not get it.
>
>  sorry.
>
> > >  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
> > >  and to get any faster you _have_ to go parallel.
> >
> > Sure, it's going to stop somewhere, but you have to be a heck of a
> > visionary to predict that it will stop _there_.
>
>  okay, i admit it: you caught me out - i'm a mad visionary.
>
>  but seriously.
>
>  it won't stop - but the price of 90nm mask charges, at approx
>  $2m, is already far too high, and the number of large chips
>  being designed is plummetting like a stone as a result - from
>  like 15,000 per year a few years ago down to ... damn, can't remember -
>  less than a hundred (i think!  don't quote me on that!)
>
>  when 90 nm was introduced, some mad fabs wanted to make 9
>  metre lenses, dude!!! until carl zeiss were called in and
>  managed to get it down to 3 metres.
>
>  and that lens is produced on a PER CHIP basis.
>
>  basically, it's about cost.

I can guarantee one thing here -- the cost, as is, is absolutely
bearable. These companies make more money doing this than they spend in
doing it, otherwise they wouldn't be in business. From an economics
perspective, this industry is very much alive and well, proven by the
fact that these companies haven't bailed out of it yet.

>  the costs of producing faster and faster uniprocessors is
>  getting out of control.
>
>  i'm not explaining things very well, but i'm trying.  too many words,
>  not concise enough, too much to explain without people misunderstanding
>  or skipping things and getting the wrong end of the stick.
>
>  argh.
>
>
> > >  and the drive for "faster", "better", "more sales" means more and more
> > >  parallelism.
> > >
> > >  it's _happening_ - and SMP ain't gonna cut it (which is why
> > >  these multi-core chips are coming out and why hyperthreading
> > >  is coming out).
> >
> > "Rah, rah, parallelism is great!" -- That's a great slogan, except...
> >
> > Users, who also happen to be the target of those sales, care about
> > _userland_ applications. And the bitter truth is that the _vast_
> > majority of userland apps are single-threaded. Why? Two reasons --
> > first, it's harder to write a multithreaded application, and second,
> > some workloads simply can't be expressed "in parallel". Your kernel
> > might (might, not will) run like a speed-demon, but the userland stuff
> > will still be lackluster in comparison.
> >
> > And that's when your slogan hits a wall, and the marketing hype dies.
> > The reality is that parallelism is something to be desired, but is not
> > always achievable.
>
>  okay: i will catch up on this bit, another time, because it is late
>  enough for me to be getting dizzy and appearing to be drunk.
>
>  this is one answer (and there are others i will write another time.
>  hint: automated code analysis tools, auto-parallelising tools, both
>  offline and realtime):

We don't need hints. We need actual performance statistics --
verifiable numbers that we can point to and say "Oh crap, we're losing."
or "Hah, we kick butt.", as the case may be.

>  watch what intel and amd do: they will support _anything_ - clutch at
>  straws - to make parallelism palable, why?  because in order to be
>  competitive - and realistically priced - they don't have any choice.

As stated earlier, I doubt they're in such dire straits as you predict.
Ultimately, the only reason why they need to advance their designs is to
be able to market it better. This means that truly innovative designs
may not be pursued because the up-front cost is too high.

There's a saying: "Let your competitor do your R&D for you."

>  plus, i am expecting the chips to be thrown out there (like
>  the X-Box 360 which has SIX hardware threads remember) and
>  the software people to quite literally _have_ to deal with it.
>
>  i expect the hardware people to go: this is the limit, this is what we
>  can do, realistically price-performance-wise: lump it, deal with it.
>
>  when intel and amd start doing that, everyone _will_ lump it.
>  and deal with it.

Hardware without software is just as useless as software without
hardware. Any argument from any side that goes along the lines of "deal
with it" can be countered in kind.

What this boils down to is that hardware people try to make their
products appealing to program to, from _both_ a speed and a usability
perspective. That's how they get mindshare.

>  ... why do you think intel is hyping support for and backing
>  hyperthreads support in XEN/Linux so much?

At the risk of stepping on some toes, I believe that hyperthreading is
going out of style, in favor of multi-core processors.

>  l.
>

In conclusion, you made claims that Linux is lagging behind. However,
such claims are rather useless without data and/or technical discussions
to back them up.

-Vadim Lobanov

Re: what's next for the linux kernel?

[D. Hazelton]

[-- Attachment #1.1: Type: text/plain, Size: 6738 bytes --]

On Monday 03 October 2005 02:31, Vadim Lobanov wrote:
> On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> > On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:
> > > On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:
> > > > On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:
> > > > > >  what if, therefore, someone comes up with an
> > > > > > architecture that is better than or improves greatly upon
> > > > > > SMP?
> > > > >
> > > > > Like NUMA?
> > > >
> > > >  yes, like numa, and there is more.
> > >
> > > The beauty of capitalization is that it makes it easier for
> > > others to read what you have to say.
> >
> >  sorry, vadim: haven't touched a shift key in over 20 years.
>
> It's not going to bite you. I promise.

You never know - someone might've rigged his keyboard to shock him 
every time the shift key was pressed :)

<snip>
> > > >  the message passing system is designed as a parallel message
> > > > bus - completely separate from the SMP and NUMA memory
> > > > architecture, and as such it is perfect for use in
> > > > microkernel OSes.
> > >
> > > You're making an implicit assumption here that it will benefit
> > > _only_ microkernel designs.
> >
> >  ah, i'm not: i just left out mentioning it :)
> >
> >  the message passing needs to be communicated down to manage
> >  threads, and also to provide a means to manage semaphores and
> >  mutexes: ultimately, support for such an architecture would
> >  work its way down to libc.
> >
> >
> >  and yes, if you _really_ didn't want a kernel in the way at all,
> > you could go embedded and just... do everything yourself.
> >
> >  or port reactos, the free software reimplementation of nt,
> >  to it, or something :)
> >
> >  *shrug*.
>
> No, for reliability and performance reasons, I very much want a
> kernel in the way. After all, kernel code is orders of magnitude
> better tuned than almost all userland code.
>
> The point I was making here is that, from what I can see, the
> current Linux architecture is quite alright in anticipation of the
> hardware that you're describing. It _could_ be better tuned for
> such hardware, sure, but so far there is no need for such work at
> this particular moment.

Wholly agreed. The arguments over the benefits of running a 
microkernel aren't ever really clear. Beyond that, I personally feel 
that the whole micro vs. mono argument is a catfight between 
academics. I'd rather have a system that works and is proven than a 
system that is bleeding edge and never truly stable.  To me this 
means a monolithic kernel - microkernels are picky at best, and can 
be highly insecure (and that means "unstable" in my book too).

<snip>
> > > >  however, as i pointed out, 90nm and approx-2Ghz is pretty
> > > > much _it_, and to get any faster you _have_ to go parallel.
> > >
> > > Sure, it's going to stop somewhere, but you have to be a heck
> > > of a visionary to predict that it will stop _there_.
> >
> >  okay, i admit it: you caught me out - i'm a mad visionary.
> >
> >  but seriously.
> >
> >  it won't stop - but the price of 90nm mask charges, at approx
> >  $2m, is already far too high, and the number of large chips
> >  being designed is plummetting like a stone as a result - from
> >  like 15,000 per year a few years ago down to ... damn, can't
> > remember - less than a hundred (i think!  don't quote me on
> > that!)
> >
> >  when 90 nm was introduced, some mad fabs wanted to make 9
> >  metre lenses, dude!!! until carl zeiss were called in and
> >  managed to get it down to 3 metres.
> >
> >  and that lens is produced on a PER CHIP basis.
> >
> >  basically, it's about cost.
>
> I can guarantee one thing here -- the cost, as is, is absolutely
> bearable. These companies make more money doing this than they
> spend in doing it, otherwise they wouldn't be in business. From an
> economics perspective, this industry is very much alive and well,
> proven by the fact that these companies haven't bailed out of it
> yet.

I have to agree. And he is also completely ignoring the fact that both 
Intel and AMD are either in the process of moving to (or have moved 
to)  a 65nm fab process - last news I saw about this said both 
facilities were running into the multi-billion dollar cost range. 
Companies worried about $2m for a mask charge wouldn't be investing 
multiple billions of dollars in new plants and a new, smaller fab 
process.

<snip>
> > > >  and the drive for "faster", "better", "more sales" means
> > > > more and more parallelism.
> > > >
> > > >  it's _happening_ - and SMP ain't gonna cut it (which is why
> > > >  these multi-core chips are coming out and why hyperthreading
> > > >  is coming out).
> > >
> > > "Rah, rah, parallelism is great!" -- That's a great slogan,
> > > except...
> > >
> > > Users, who also happen to be the target of those sales, care
> > > about _userland_ applications. And the bitter truth is that the
> > > _vast_ majority of userland apps are single-threaded. Why? Two
> > > reasons -- first, it's harder to write a multithreaded
> > > application, and second, some workloads simply can't be
> > > expressed "in parallel". Your kernel might (might, not will)
> > > run like a speed-demon, but the userland stuff will still be
> > > lackluster in comparison.
> > >
> > > And that's when your slogan hits a wall, and the marketing hype
> > > dies. The reality is that parallelism is something to be
> > > desired, but is not always achievable.
> >
> >  okay: i will catch up on this bit, another time, because it is
> > late enough for me to be getting dizzy and appearing to be drunk.
> >
> >  this is one answer (and there are others i will write another
> > time. hint: automated code analysis tools, auto-parallelising
> > tools, both offline and realtime):
>
> We don't need hints. We need actual performance statistics --
> verifiable numbers that we can point to and say "Oh crap, we're
> losing." or "Hah, we kick butt.", as the case may be.

Hear, hear!  I'm still working my way through the source tree and 
learning the general layout and functionality of the various bits, 
but in just a pair of months of being on this list I can attest to 
the fact that one thing all developers seem to ask for is statistics.

<snip>
> At the risk of stepping on some toes, I believe that hyperthreading
> is going out of style, in favor of multi-core processors.

Agreed. And multi-core processors aren't really new technology - there 
have been multi-core designs out for a while, but those were usually 
low production "research" chips.

DRH

[-- Attachment #1.2: 0xA6992F96300F159086FF28208F8280BB8B00C32A.asc --]
[-- Type: application/pgp-keys, Size: 1365 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: what's next for the linux kernel?

[Giuseppe Bilotta]

On Mon, 3 Oct 2005 02:53:02 +0100, Luke Kenneth Casson Leighton wrote:

> On Sun, Oct 02, 2005 at 06:20:38PM -0700, Vadim Lobanov wrote:
>> The beauty of capitalization is that it makes it easier for others to
>> read what you have to say.
>  
>  sorry, vadim: haven't touched a shift key in over 20 years.
> 

[snip]

>  *sigh* - i _so_ hope we don't need degrees in physics to program
>  them...

[snip]

>  ah, i'm not: i just left out mentioning it :)

I'd *love* a keyboard layout where * _ : ) are accesible without
shift! Can you send me yours?

-- 
Giuseppe "Oblomov" Bilotta

Hic manebimus optime

Re: what's next for the linux kernel?

[Nix]

On 3 Oct 2005, Giuseppe Bilotta prattled cheerily:
> I'd *love* a keyboard layout where * _ : ) are accesible without
> shift! Can you send me yours?

<http://www.maltron.co.uk/images/press/maltron-ergonomic-english-trackball-tq-hr1.jpg>

(downside: cost. Upside: feels lovely.)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Mon, Oct 03, 2005 at 02:53:02AM +0100, Luke Kenneth Casson Leighton wrote:
>  sorry, vadim: haven't touched a shift key in over 20 years.

Except to type that ':' I suspect.

How about learning to use that shift key so that everyone else that
reads your typing don't have to spend so much time working it out when
proper syntax would have made it simpler.  It may save you 1% of your
time typing it, but you cost thousands of people much more as a result.
net loss for the world as a whole.

>  ah, i'm not: i just left out mentioning it :)
> 
>  the message passing needs to be communicated down to manage
>  threads, and also to provide a means to manage semaphores and
>  mutexes: ultimately, support for such an architecture would
>  work its way down to libc.
> 
>  and yes, if you _really_ didn't want a kernel in the way at all, you
>  could go embedded and just... do everything yourself.
> 
>  or port reactos, the free software reimplementation of nt,
>  to it, or something :)

Microkernel, message passing, blah blah blah.  Does GNU Hurd actually
run fast yet?  I think it exists finally and works (at least mostly) but
how does the performance compare?

Most of your arguments seem like a repeat of more acedemic theories most
of which have not been used in a real system where performance running
average software was important, at least I never heard of them.  Not
that that necesarily means much, other than they can't have gained much
popularity.

>  it won't stop - but the price of 90nm mask charges, at approx
>  $2m, is already far too high, and the number of large chips
>  being designed is plummetting like a stone as a result - from
>  like 15,000 per year a few years ago down to ... damn, can't remember -
>  less than a hundred (i think!  don't quote me on that!)

Hmm, so if we guess it might take 10 masks per processor type over it's
life time as they change features and such, that's still less than 1% of
the cost of the FAB in the first place.  I agree with the person that
said intel/AMD/company probably don't care much, as long as their
engineers make really darn sure that the mask is correct when they go to
make one.

>  okay: i will catch up on this bit, another time, because it is late
>  enough for me to be getting dizzy and appearing to be drunk.
> 
>  this is one answer (and there are others i will write another time.
>  hint: automated code analysis tools, auto-parallelising tools, both
>  offline and realtime):
> 
>  watch what intel and amd do: they will support _anything_ - clutch at
>  straws - to make parallelism palable, why?  because in order to be
>  competitive - and realistically priced - they don't have any choice.
> 
>  plus, i am expecting the chips to be thrown out there (like
>  the X-Box 360 which has SIX hardware threads remember) and
>  the software people to quite literally _have_ to deal with it.

Hey I like parallel processing.  i think it's neat, and I have often
made some of my own tools multithreaded just because I found it could be
done for the task, and I often find multithreaded simpler to code (I
seem to be a bit of a weirdo that way) for certain tasks.

>  i expect the hardware people to go: this is the limit, this is what we
>  can do, realistically price-performance-wise: lump it, deal with it.
> 
>  when intel and amd start doing that, everyone _will_ lump it.
>  and deal with it.
> 
>  ... why do you think intel is hyping support for and backing
>  hyperthreads support in XEN/Linux so much?

Ehm, because intel has it and their P4 desperately needs help to gain
any performance it can until they get the Pentium-M based desktop chips
finished with multiple cores, and of course because AMD doesn't have it.
Seem like good reasons for intel to try and push it.

Len Sorensen

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

> Hmm, so if we guess it might take 10 masks per processor type over it's
> life time as they change features and such, that's still less than 1% of
> the cost of the FAB in the first place.  I agree with the person that
> said intel/AMD/company probably don't care much, as long as their
> engineers make really darn sure that the mask is correct when they go to
> make one.
 
 you elaborate, therefore, on my point.

 anyone else, therefore, cannot hope to compete or even enter the
 market, at 90nm.

 which is why the first VIA eden processors maxed out at 800mhz (i'm
 guessing they were a 0.13micron and therefore 2.5 volts)

> >  ... why do you think intel is hyping support for and backing
> >  hyperthreads support in XEN/Linux so much?
> 
> Ehm, because intel has it and their P4 desperately needs help to gain
> any performance it can until they get the Pentium-M based desktop chips
> finished with multiple cores, and of course because AMD doesn't have it.
> Seem like good reasons for intel to try and push it.
 
 you lend weight to my earlier points: the push is to
 drive the engineers towards less gates on the excuse of
 cart-before-horsing the market with their "performance / watt"
 metrics, such that if 0.65nm comes off it's less painful
 and not too much of a jump, and they aim for more parallel
 processing (multiple cores).

 current  : 200 million gates with 90nm at 1.65 volt
 estimated: 40 million gates with 65nm at 1.1 volt
 estimated: 1 million gates with 45nm at 0.9 volt.

 the "off" voltage of a silicon germanium transistor is 0.8 volts.

 at 45nm the current leakage is so insane that the heat
 dissipation, through the oxide layer which covers the chip,
 ends up blowing the chip up.

 trouble.

 l.

Re: what's next for the linux kernel?

[linux-os (Dick Johnson)]

On Tue, 4 Oct 2005, Luke Kenneth Casson Leighton wrote:

>> Hmm, so if we guess it might take 10 masks per processor type over it's
>> life time as they change features and such, that's still less than 1% of
>> the cost of the FAB in the first place.  I agree with the person that
>> said intel/AMD/company probably don't care much, as long as their
>> engineers make really darn sure that the mask is correct when they go to
>> make one.
>
> you elaborate, therefore, on my point.
>
> anyone else, therefore, cannot hope to compete or even enter the
> market, at 90nm.
>
> which is why the first VIA eden processors maxed out at 800mhz (i'm
> guessing they were a 0.13micron and therefore 2.5 volts)
>
>>>  ... why do you think intel is hyping support for and backing
>>>  hyperthreads support in XEN/Linux so much?
>>
>> Ehm, because intel has it and their P4 desperately needs help to gain
>> any performance it can until they get the Pentium-M based desktop chips
>> finished with multiple cores, and of course because AMD doesn't have it.
>> Seem like good reasons for intel to try and push it.
>
> you lend weight to my earlier points: the push is to
> drive the engineers towards less gates on the excuse of
> cart-before-horsing the market with their "performance / watt"
> metrics, such that if 0.65nm comes off it's less painful
> and not too much of a jump, and they aim for more parallel
> processing (multiple cores).
>
> current  : 200 million gates with 90nm at 1.65 volt
> estimated: 40 million gates with 65nm at 1.1 volt
> estimated: 1 million gates with 45nm at 0.9 volt.
>
> the "off" voltage of a silicon germanium transistor is 0.8 volts.
>
> at 45nm the current leakage is so insane that the heat
> dissipation, through the oxide layer which covers the chip,
> ends up blowing the chip up.
>
> trouble.
>
> l.

Since the voltage goes down as the speed increases, an
industry joke has been that at 0 volts you can get the
speed that you want. Unfortunately, the required infinite
current is a bit hard to manage. This brings up the
configuration change that has been in the works for
some time, current-mode logic. Just like TTL gave way
to ECL to obtain two orders of magnitude increase in
random-logic speed, there will be a similar increase once
current rather than voltage is used for logic levels.
And, it's not absolute current, either but delta-current
which will define a logic state.

The problems with reducing capacity end up being exacerbated
when logic states are stored in the very capacity that is
being reduced. Eventually there is little difference between
"logic" and "noise". This brings up the new science of
"statistical logic" which has yet to make its way into
microprocessors, but soon will once the quantization noise
becomes a factor.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Tue, Oct 04, 2005 at 01:53:54PM +0100, Luke Kenneth Casson Leighton wrote:
>  you elaborate, therefore, on my point.
> 
>  anyone else, therefore, cannot hope to compete or even enter the
>  market, at 90nm.
> 
>  which is why the first VIA eden processors maxed out at 800mhz (i'm
>  guessing they were a 0.13micron and therefore 2.5 volts)

Sticking with 0.13 also seems to make for a better embedded processor
since it seems when you go to 0.09 or smaller it becomes nearly
impossible to run at high and low temperatures which is what some
embedded applications want (like -40 to +85C).  We had to change compact
flash suppliers when our supplier went to 90nm since they said their new
process couldn't work reliably at industrial temperature anymore.  If
VIA wants the eden to run at a wide temperature range, it appears they
are better off sticking to a larger process and keeping the cpu speed
down to something reasonable.  I imagine at some point someone will
manage to make a 90nm chip that does handle bigger temperature ranges
but I haven't seen one yet myself.

>  you lend weight to my earlier points: the push is to
>  drive the engineers towards less gates on the excuse of
>  cart-before-horsing the market with their "performance / watt"
>  metrics, such that if 0.65nm comes off it's less painful
>  and not too much of a jump, and they aim for more parallel
>  processing (multiple cores).

If that was the goal the x86 architecture should be dumped.  It spends
too many gates converting x86 junk into something reasonable to execute.
People don't appear very eager to dump the x86 unfortunately.  Something
to do with backwards compatibility and such.

>  current  : 200 million gates with 90nm at 1.65 volt
>  estimated: 40 million gates with 65nm at 1.1 volt
>  estimated: 1 million gates with 45nm at 0.9 volt.

A 1 million gate chip at 45nm would be rather tiny.  The yield would
probably be amazingly good.  Of course if it does give off a lot of heat
you still have the problem of how to get rid of the heat given it is
focused in a very very small space.  Of course just reducing the size of
the cache on intel's chips to something sane would reduce the gate count
enourmously.  But that won't happen until they make a more efficient
chip.

>  the "off" voltage of a silicon germanium transistor is 0.8 volts.
> 
>  at 45nm the current leakage is so insane that the heat
>  dissipation, through the oxide layer which covers the chip,
>  ends up blowing the chip up.

Unless someone finds a way to reduce the leakage.  It is worth a lot of
money to some companies to solve that problem after all.

Len Sorensen

Re: what's next for the linux kernel?

[Bill Davidsen]

Lennart Sorensen wrote:
> On Tue, Oct 04, 2005 at 01:53:54PM +0100, Luke Kenneth Casson Leighton wrote:
> 

>> at 45nm the current leakage is so insane that the heat
>> dissipation, through the oxide layer which covers the chip,
>> ends up blowing the chip up.
> 
> 
> Unless someone finds a way to reduce the leakage.  It is worth a lot of
> money to some companies to solve that problem after all.

That's one way, the other is to find a way to cool such a chip. I see 
references to diamond substrate from time to time, good thermal 
conductor. So are other carbon forms, Fullerines, etc.

Clearly reducing leakage is the optimal solution, "deal with the heat" 
is the other.

Re: what's next for the linux kernel?

[Gene Heskett]

On Tuesday 04 October 2005 08:53, Luke Kenneth Casson Leighton wrote:
[...]
> the "off" voltage of a silicon germanium transistor is 0.8
volts.

Isn't that a bit contradictory, Luke?.  Or is there a new process that
mixes the two basic materials for making semiconductors from?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: what's next for the linux kernel?

[Horst von Brand]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Sun, Oct 02, 2005 at 04:37:52PM -0700, Vadim Lobanov wrote:
> > >  what if, therefore, someone comes up with an architecture that is
> > >  better than or improves greatly upon SMP?

> > Like NUMA?

>  yes, like numa, and there is more.
> 
>  i had the honour to work with someone who came up with a radical
>  enhancement even to _that_.

Any papers to look at?

>  basically the company has implemented, in hardware (a nanokernel),

A nanokernel is a piece of software in my book?

>                                                                     some
>  operating system primitives, such as message passing (based on a
>  derivative by thompson of the "alice" project from plessey, imperial and
>  manchester university in the mid-80s), hardware cache line lookups
>  (which means instead of linked list searching, the hardware does it for
>  you in a single cycle), stuff like that.

Single CPU cycle for searching data in memory? Impossible.

>  the message passing system is designed as a parallel message bus -
>  completely separate from the SMP and NUMA memory architecture, and as
>  such it is perfect for use in microkernel OSes.

Something must shuffle the data from "regular memory" into "message
memory", so I bet that soon becomes the bottleneck. And the duplicate data
paths add to the cost, money that could be spent on making memory access
faster, so...

>  (these sorts of things are unlikely to make it into the linux kernel, no
>  matter how much persuasion and how many patches they would write).

Your head would apin when looking at how fast this gets into Linux if there
were such machines around, and it is worth it.

>  _however_, a much _better_ target would be to create an L4 microkernel
>  on top of their hardware kernel.

Not yet another baroque CISC design, this time around with 1/3 of an OS in
it!

>  this company's hardware is kinda a bit difficult for most people to get
>  their heads round: it's basically parallelised hardware-acceleration for
>  operating systems, and very few people see the point in that.

Perhaps most people that don't see the point do have a point?

>  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
>  and to get any faster you _have_ to go parallel.

Sorry, all this has been doomsayed (with different numbers) from 1965 or
so.

>  and the drive for "faster", "better", "more sales" means more and more
>  parallelism.

Right.

>  it's _happening_ - and SMP ain't gonna cut it (which is why
>  these multi-core chips are coming out and why hyperthreading
>  is coming out).

Hyperthreading and multi-core /are/ SMP, just done a bit differently.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

On Sun, 02 Oct 2005 22:12:38 EDT, Horst von Brand said:

> >                                                                     some
> >  operating system primitives, such as message passing (based on a
> >  derivative by thompson of the "alice" project from plessey, imperial and
> >  manchester university in the mid-80s), hardware cache line lookups
> >  (which means instead of linked list searching, the hardware does it for
> >  you in a single cycle), stuff like that.
> 
> Single CPU cycle for searching data in memory? Impossible.

Well... if it was content-addressable RAM similar to what's already used for
the hardware TLB's and the like - just that it's one thing to make a 32 or 256
location content-addressable RAM, and totally another to have multiple megabytes
of the stuff. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 12:32:35PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Sun, 02 Oct 2005 22:12:38 EDT, Horst von Brand said:
> 
> > >                                                                     some
> > >  operating system primitives, such as message passing (based on a
> > >  derivative by thompson of the "alice" project from plessey, imperial and
> > >  manchester university in the mid-80s), hardware cache line lookups
> > >  (which means instead of linked list searching, the hardware does it for
> > >  you in a single cycle), stuff like that.
> > 
> > Single CPU cycle for searching data in memory? Impossible.
> 
> Well... if it was content-addressable RAM similar to what's already used for
> the hardware TLB's and the like - just that it's one thing to make a 32 or 256
> location content-addressable RAM, and totally another to have multiple megabytes
> of the stuff. :)

 aspex microelectronics 4096 2-bit massively parallel SIMD
 processor (does 1 terabit-ops / sec @ 250mhz which sounds a
 lot until you try to do FPU emulation on it).

 each 2-bit processor has 256 bits of content-addressable memory,
 which can be 8-bit, 16-bit or 32-bit addressed (to make 4096 parallel
 memory searches - in a single cycle).

 absolutely friggin blindingly fast for certain issues (video
 processing, certain kinds of audio processing - e.g. FFTs,
 XML and HTTP parsing), and pissed all over for others such
 as doing floating point arithmetic.

 but anyway: that's a side issue.  thanks for reminding me about CAM,
 valdis.

 l.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Mon, 03 Oct 2005 01:54:00 BST, Luke Kenneth Casson Leighton said:

>  in the mid-80s), hardware cache line lookups (which means
>  instead of linked list searching, the hardware does it for
>  you in a single cycle), stuff like that.

OK.. I'll bite.  How do you find the 5th or 6th entry in the linked list,
when only the first entry is in cache, in a single cycle, when a cache line
miss is more than a single cycle penalty, and you have several "These are not
the droids you're looking for" checks and go on to the next entry - and do it
in one clock cycle?

Now, it's really easy to imagine an execution unit that will execute this
as a single opcode, and stall until complete.  Of course, this only really helps
if you have multiple execution units - which is what hyperthreading and
multi-core and all that is about.  And guess what - it's not news...

The HP2114 and DEC KL10/20 were able to dereference a chain of indirect bits
back in the 70's (complete with warnings that hardware wedges could occur if
an indirect reference formed a loop or pointed at itself). Whoops. :)

And all the way back in 1964, IBM disk controllers were able to do some rather
sophisticated offloading of "channel control words" (amazing what you could do
with 'Search ID Equal', 'Transfer In-Channel' (really a misnamed branch
instruction), and self-modifying CCWs).  But even then, they understood that
it was only a win if you could go do other stuff when you waited....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Rik van Riel]

On Sun, 2 Oct 2005, Valdis.Kletnieks@vt.edu wrote:

> OK.. I'll bite.  How do you find the 5th or 6th entry in the linked 
> list, when only the first entry is in cache, in a single cycle, when a 
> cache line miss is more than a single cycle penalty, and you have 
> several "These are not the droids you're looking for" checks and go on 
> to the next entry - and do it in one clock cycle?

A nice saying from the last decade comes to mind:

"If you can do all that in one cycle, your cycles are too long."

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Alan Cox]

On Sul, 2005-10-02 at 22:55 -0400, Valdis.Kletnieks@vt.edu wrote:
> The HP2114 and DEC KL10/20 were able to dereference a chain of indirect bits
> back in the 70's (complete with warnings that hardware wedges could occur if
> an indirect reference formed a loop or pointed at itself). 

The KL10 has an 8 way limit. The PDP-6 didn't but then it also lacked an
MMU.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 10:55:49PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Mon, 03 Oct 2005 01:54:00 BST, Luke Kenneth Casson Leighton said:
> 
> >  in the mid-80s), hardware cache line lookups (which means
> >  instead of linked list searching, the hardware does it for
> >  you in a single cycle), stuff like that.
> 
> OK.. I'll bite.  How do you find the 5th or 6th entry in the linked list,
> when only the first entry is in cache, in a single cycle, when a cache line
> miss is more than a single cycle penalty, and you have several "These are not
> the droids you're looking for" checks and go on to the next entry - and do it
> in one clock cycle?
 
  i was not privy to the design discussions: unfortunately i was only
  given brief conclusions and hints by the designer.

  my guess is that yes, as the later messages in this thread
  hint at, CAM is probably the key: 256 blocks of 32-bit CAM,
  something like that.

  CAM is known to help dramatically decrease execution time
  by orders of magnitude in linked list algorithms such as
  searching and sorting, esp. where each CAM cell has built-in
  processing, like the aspex.net massively-deep SIMD architecture has.

  l.

Re: what's next for the linux kernel?

[Sonny Rao]

On Mon, Oct 03, 2005 at 01:54:00AM +0100, Luke Kenneth Casson Leighton wrote:
<snip>
>  this company's hardware is kinda a bit difficult for most people to get
>  their heads round: it's basically parallelised hardware-acceleration for
>  operating systems, and very few people see the point in that.

Obviously, we are all clueless morons.

<snip>

>  so.
> 
>  this is a heads-up.
> 
>  what you choose to do with this analysis is up to you.
> 
>  l.

Roll around on the floor while violently laughing for a while?

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 01:03:48AM -0400, Sonny Rao wrote:
> Roll around on the floor while violently laughing for a while?

 _excellent_!  can we watch?  where's the mp4 :)

Re: what's next for the linux kernel?

[Sonny Rao]

On Mon, Oct 03, 2005 at 10:12:26PM +0100, Luke Kenneth Casson Leighton wrote:
> On Mon, Oct 03, 2005 at 01:03:48AM -0400, Sonny Rao wrote:
> > Roll around on the floor while violently laughing for a while?
> 
>  _excellent_!  can we watch?  where's the mp4 :)

It'll be out there shortly... ;-P

But realistically, how can I take you seriously?  

You come onto the Linux kernel mailing list, talk about hardware
implementations that are not generally known or available while
insulting the populace by claiming they would not understand
anyway,  then insult the kernel developers by proclaiming their hard work
is "daft" and that the design is going to be "_really_ outdated,
really soon" without much explanation, go on to make wildly
generalized comments about processor frequency scaling with little
real insight or explanation of the issues, and apparently like making
long meandering posts even more difficult to read by not using capital
letters. 

I also notice you suddenly decided to act morally superior to Martin
by taking "a leaf from the great rusty russell's book " and dealing
with "immature and out-of-line comments" in a "professional way."
Way to go...

Anyway, others have posted far more mature responses than my obvious
flamebait... apologies for the extra noise.

Sonny

Re: what's next for the linux kernel?

[Alan Cox]

On Llu, 2005-10-03 at 01:54 +0100, Luke Kenneth Casson Leighton wrote:
>  the message passing system is designed as a parallel message bus -
>  completely separate from the SMP and NUMA memory architecture, and as
>  such it is perfect for use in microkernel OSes.

I've got one of those. It has the memory attached. Makes a fantastic
message bus and has a really long queue. Also features shortcuts for
messages travelling between processors in short order cache to cache.
Made by AMD and Intel.

>  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
>  and to get any faster you _have_ to go parallel.

We do 512 processors passably now. Thats a lot of cores and more than
the commodity computing people can wire to memory subsystems at a price
people will pay. 

Besides which you need to take it up with the desktop people really. Its
their apps that use most of the processor power and will benefit most
from parallelising and efficiency work. 

Alan

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 08:18:40PM +0100, Alan Cox wrote:
> On Llu, 2005-10-03 at 01:54 +0100, Luke Kenneth Casson Leighton wrote:
> >  the message passing system is designed as a parallel message bus -
> >  completely separate from the SMP and NUMA memory architecture, and as
> >  such it is perfect for use in microkernel OSes.
> 
> I've got one of those. It has the memory attached. Makes a fantastic
> message bus and has a really long queue. Also features shortcuts for
> messages travelling between processors in short order cache to cache.
> Made by AMD and Intel.
 
 made?  _cool_.  actual hardware.  new knowledge for me.  do you know
 of any online references, papers or stuff?  [btw just to clarify:
 you're saying you have a NUMA bus or you're saying you have an
 augmented SMP+NUMA+separate-parallel-message-passing-bus er .. thing]


> >  however, as i pointed out, 90nm and approx-2Ghz is pretty much _it_,
> >  and to get any faster you _have_ to go parallel.
> 
> We do 512 processors passably now. 

 wild.

> Thats a lot of cores and more than
> the commodity computing people can wire to memory subsystems at a price
> people will pay. 

 oops.

 whereas, would you see it more reasonable for a commodity-level
 chip to be something like 32- or even 64- ultra-RISC cores of
 between 5000 and 10,000 gates each, resulting in a processor
 of about 50% cache memory and 50% processing plus associated
 parallel bus architecture at around 1 million gates?

 running at oh say 1ghz or with careful design effort focussed on the
 RISC cores maybe even 2ghz, resulting in 128 total GigaOps if you
 go for 64 cpus @ 2ghz.  that's a friggin lot of processing power
 for a 1m gates processor!!

 (hey, see, i can learn to use the shift key to highlight the keyword)

 such a chip, in 90nm, would be approx $USD 20 in mass production.

 small, good heat distribution, probably too many pins, probably
 need some DRAM memory stamped upside down on top of the die,
 instead of off-chip [putting DRAM and transistors on the same die
 is a frequent and costly mistake: the yields are terrible].

 putting DRAM upside down on top of a die and then hitting it
 with a hammer (literally) is a frequently used technique to
 avoid the problem.


> Besides which you need to take it up with the desktop people really. 

 i'm sort-of drafting a reply to rik's point in my head (no it's not
 reiterations of things already said) and yes it involves legacy apps,
 badly written apps, desktop focus etc.

 server-side, yep, fine, 32-way, use it all, got it, heck,
 even my apache server running on a P200 w/64mb RAM runs more
 processes than that.

 l.

Re: what's next for the linux kernel?

[Alan Cox]

On Llu, 2005-10-03 at 22:07 +0100, Luke Kenneth Casson Leighton wrote:
>  made?  _cool_.  actual hardware.  new knowledge for me.  do you know
>  of any online references, papers or stuff?  [btw just to clarify:
>  you're saying you have a NUMA bus or you're saying you have an
>  augmented SMP+NUMA+separate-parallel-message-passing-bus er .. thing]

Its a standard current Intel feature. See "mwait" in the processor
manual. The CPUs are also smart enough to do cache to cache transfers.
No special hardware no magic.

And unless I want my messages to cause interrupts and wake events (in
which case the APIC does it nicely) then any locked operation on memory
will do the job just fine. I don't need funky hardware on a system. The
first point I need funky hardware is between boards and that isn't
consumer any more.

Alan

Re: what's next for the linux kernel?

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> On Llu, 2005-10-03 at 22:07 +0100, Luke Kenneth Casson Leighton wrote:
> >  made?  _cool_.  actual hardware.  new knowledge for me.  do you know
> >  of any online references, papers or stuff?  [btw just to clarify:
> >  you're saying you have a NUMA bus or you're saying you have an
> >  augmented SMP+NUMA+separate-parallel-message-passing-bus er .. thing]
> 
> Its a standard current Intel feature. See "mwait" in the processor
> manual. The CPUs are also smart enough to do cache to cache transfers.
> No special hardware no magic.

It's unfortunately useless for anything but kernels right now because
Intel has disabled it in ring 3 (and AMD doesn't support it yet)
And the only good use the kernel found for it so far is fast wakeup
from the idle loop.

> And unless I want my messages to cause interrupts and wake events (in
> which case the APIC does it nicely) then any locked operation on memory
> will do the job just fine. I don't need funky hardware on a system. The
> first point I need funky hardware is between boards and that isn't
> consumer any more.

Firewire + CLFLUSH should do the job.

-Andi
 

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

On Mon, 03 Oct 2005 22:07:22 BST, Luke Kenneth Casson Leighton said:

>  whereas, would you see it more reasonable for a commodity-level
>  chip to be something like 32- or even 64- ultra-RISC cores of
>  between 5000 and 10,000 gates each, resulting in a processor
>  of about 50% cache memory and 50% processing plus associated
>  parallel bus architecture at around 1 million gates?

Read your history - especially IBM's 801 chipset, which became the RT,
and why they then replaced that with the Power architecture...

>  running at oh say 1ghz or with careful design effort focussed on the
>  RISC cores maybe even 2ghz, resulting in 128 total GigaOps if you
>  go for 64 cpus @ 2ghz.  that's a friggin lot of processing power
>  for a 1m gates processor!!

Good.  Were you planning to run the ucLinux branch on this, or include all
the pieces needed to support virtual memory?  And do it inside that 10K gate
budget, too (hint - how many gates will you burn just doing a TLB big enough
to get the performance of mapping a virtual->real address to be good enough?)

You might want to read up on all the fun that IBM went through in designing
memory subsystems that can keep even the Power4 and Power5 chipsets fed too,
or the interesting stuff that SGI has to do to allow 64/128/512 processors
to beat up on a memory system - I'm sure there's some pretty high gate count
involved there..

If you're doing 64 10K-gate cores, you've blown 64% of your 1M gate budget.
You've got only 320K gates left to build cache *and* virtual memory support to
make that 1M gate budget.  And yes, you need a cache, as IBM found out on
their RT processor.....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Martin J. Bligh]

--Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote (on Monday, October 03, 2005 00:05:45 +0100):

> On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
>> On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:
>> 
>> > and, what is the linux kernel?
>> > 
>> > it's a daft, monolithic design that is suitable and faster on
>> > single-processor systems, and that design is going to look _really_
>> > outdated, really soon.
>> 
>> Linux already has a number of scalable SMP synchronisation
>> mechanisms. 
> 
>  ... and you are tied in to the decisions made by the linux kernel
>  developers.

Yes. As are the rest of us. So if you want to implement something 
different, that's your perogative. So feel free to go do it 
somewhere else, and quit whining on this list. 

We are not your implementation bitches. If you think it's such a great
idea, do it yourself.

M.

 

Re: what's next for the linux kernel?

[Randy.Dunlap]

On Sun, 02 Oct 2005 17:04:51 -0700 Martin J. Bligh wrote:

> --Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote (on Monday, October 03, 2005 00:05:45 +0100):
> 
> > On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
> >> On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:
> >> 
> >> > and, what is the linux kernel?
> >> > 
> >> > it's a daft, monolithic design that is suitable and faster on
> >> > single-processor systems, and that design is going to look _really_
> >> > outdated, really soon.
> >> 
> >> Linux already has a number of scalable SMP synchronisation
> >> mechanisms. 
> > 
> >  ... and you are tied in to the decisions made by the linux kernel
> >  developers.
> 
> Yes. As are the rest of us. So if you want to implement something 
> different, that's your perogative. So feel free to go do it 
> somewhere else, and quit whining on this list. 
> 
> We are not your implementation bitches. If you think it's such a great
> idea, do it yourself.

IOW, -ENOPATCH.  where's your patch?

---
~Randy
You can't do anything without having to do something else first.
-- Belefant's Law

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 05:14:57PM -0700, Randy.Dunlap wrote:

> IOW, -ENOPATCH.  where's your patch?
 
 most of the relevant work has already been done (and not by
 me): i invite you to consider searching with google for l4ka,
 l4linux and oskit, or simply going to the web site l4linux.org
 and l4ka.org.

 the code for oskit has been available for some years, now,
 and is regularly maintained.  the l4linux people have had to
 make some significant modifications to it (oskit), and also
 to grub, and libstdc++, and pretty much everything else under
 the sun and, it's all there, for the approx 100mb downloading.

 l.

Re: what's next for the linux kernel?

[Meelis Roos]

LKCL>  the code for oskit has been available for some years, now,
LKCL>  and is regularly maintained.  the l4linux people have had to

My experience with oskit (trying to let students use it for OS course
homework) is quite ... underwhelming. It works as long as you try to use
it exactly like the developers did and breaks on a slightest sidestep
from that road. And there's not much documentation so it's hard to learn
where that road might be.

Switched to Linux/BSD code hacking with students, the code that actually
works.

YMMV.

-- 
Meelis Roos

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Mon, Oct 03, 2005 at 10:50:00AM +0300, Meelis Roos wrote:
> LKCL>  the code for oskit has been available for some years, now,
> LKCL>  and is regularly maintained.  the l4linux people have had to
> 
> My experience with oskit (trying to let students use it for OS course
> homework) is quite ... underwhelming. It works as long as you try to use
> it exactly like the developers did and breaks on a slightest sidestep
> from that road. And there's not much documentation so it's hard to learn
> where that road might be.
> 
> Switched to Linux/BSD code hacking with students, the code that actually
> works.

Can oskit be worse than nachos where the OS ran outside the memory space
and cpu with only applications being inside the emulated mips processor?
Made some things much too easy to do, and other things much to hard
(like converting an address from user space to kernel space an accessing
it, which should be easy, but was hard).

I suspect most 'simple' OS teaching tools are awful.  Of course writing
a complete OS from scratch is a serious pain and makes debuging much
harder than if you can do your work on top of a working OS that can
print debug messages.

Len Sorensen

Re: what's next for the linux kernel?

[linux-os (Dick Johnson)]

On Mon, 3 Oct 2005, Lennart Sorensen wrote:

> On Mon, Oct 03, 2005 at 10:50:00AM +0300, Meelis Roos wrote:
>> LKCL>  the code for oskit has been available for some years, now,
>> LKCL>  and is regularly maintained.  the l4linux people have had to
>>
>> My experience with oskit (trying to let students use it for OS course
>> homework) is quite ... underwhelming. It works as long as you try to use
>> it exactly like the developers did and breaks on a slightest sidestep
>> from that road. And there's not much documentation so it's hard to learn
>> where that road might be.
>>
>> Switched to Linux/BSD code hacking with students, the code that actually
>> works.
>
> Can oskit be worse than nachos where the OS ran outside the memory space
> and cpu with only applications being inside the emulated mips processor?
> Made some things much too easy to do, and other things much to hard
> (like converting an address from user space to kernel space an accessing
> it, which should be easy, but was hard).
>
> I suspect most 'simple' OS teaching tools are awful.  Of course writing
> a complete OS from scratch is a serious pain and makes debuging much
> harder than if you can do your work on top of a working OS that can
> print debug messages.
>
> Len Sorensen
> -

But the first thing you must do in a 'roll-your-own' OS is to make
provisions to write text to (sometimes a temporary) output device
and get some input from same. Writing such basic stuff is getting
harder because many embedded systems don't have UARTS, screen-cards,
keyboards, or any useful method of doing I/O. This is where an
existing OS (Like Linux) can help you get some I/O running, perhaps
through a USB bus. You debug and make it work as a Linux
Driver, then you link the working stuff into your headless CPU
board.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: what's next for the linux kernel?

[Jon Masters]

On 10/3/05, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>
> On Mon, 3 Oct 2005, Lennart Sorensen wrote:

> > I suspect most 'simple' OS teaching tools are awful.  Of course writing
> > a complete OS from scratch is a serious pain and makes debuging much
> > harder than if you can do your work on top of a working OS that can
> > print debug messages.

> But the first thing you must do in a 'roll-your-own' OS is to make
> provisions to write text to (sometimes a temporary) output device
> and get some input from same.

Indeed. I started work on a microkernel for a final year University
project. Didn't get very far beyond minimal memory management and a
vague handy-wavy concept of a process in the end as it's easy to get
unstuck figuring out random blackbox hardware. Makes you respect some
of these people who really figured it out for real and got it working.

> Writing such basic stuff is getting harder because many embedded
> systems don't have UARTS, screen-cards, keyboards, or any useful
> method of doing I/O.

It's easier now that we have a growing number of cheaper ARM/PPC
boards on the market. But in order to do much of this, you really need
a hardware debugger. In my case, I tried to do this on an Apple
powerbook but once you've broken the BAT/page mapping for your
framebuffer you're rapidly running out of ways of debugging, e.g. a
VM. It's difficult enough even with a UART, or an LED, or whatever.

> This is where an existing OS (Like Linux) can help you get some I/O
> running, perhaps through a USB bus. You debug and make it work
> as a Linux Driver, then you link the working stuff into your headless
> CPU board.

A lot of people end up doing that - I've heard of some interesting
stories which I'm sure aren't widespread. One case, the guy had
basically bolted a small realtime module on to Linux (not really quite
like RTLinux) but had been able to do a lot of testing through
existing APIs. Another trick is to write as much as you can to sit
right atop the existing firmware - OpenFirmware, U-Boot, whatever and
perhaps even forgo trying to handle exceptions/VM for yourself in the
beginning.

Jon.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 02:08:58PM -0400, Lennart Sorensen wrote:
> On Mon, Oct 03, 2005 at 10:50:00AM +0300, Meelis Roos wrote:
> > LKCL>  the code for oskit has been available for some years, now,
> > LKCL>  and is regularly maintained.  the l4linux people have had to
> > 
> > My experience with oskit (trying to let students use it for OS course
> > homework) is quite ... underwhelming. It works as long as you try to use
> > it exactly like the developers did and breaks on a slightest sidestep
> > from that road. And there's not much documentation so it's hard to learn
> > where that road might be.

analysis, verification, debugging and adoption of oskit by
the linux kernel maintainers would help enormously there,
i believe, which is why i invited the kernel maintainers to
give it some thought.

there are other reasons: not least is that oskit _is_ the
linux kernel source code - with the kernel/* bits removed and
the device drivers and support infrastructure remaining.


so the developers who split the linux source code out into oskit did
not, in your opinion and experience, meelis, do a very good job: so
educate them and tell them how to do it better.

l.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 05:04:51PM -0700, Martin J. Bligh wrote:
> --Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote (on Monday, October 03, 2005 00:05:45 +0100):
> 
> > On Sun, Oct 02, 2005 at 05:05:42PM -0400, Rik van Riel wrote:
> >> On Sun, 2 Oct 2005, Luke Kenneth Casson Leighton wrote:
> >> 
> >> > and, what is the linux kernel?
> >> > 
> >> > it's a daft, monolithic design that is suitable and faster on
> >> > single-processor systems, and that design is going to look _really_
> >> > outdated, really soon.
> >> 
> >> Linux already has a number of scalable SMP synchronisation
> >> mechanisms. 
> > 
> >  ... and you are tied in to the decisions made by the linux kernel
> >  developers.
> 
> Yes. As are the rest of us. So if you want to implement something 
> different, that's your perogative. So feel free to go do it 
> somewhere else, and quit whining on this list. 
> 
> We are not your implementation bitches. If you think it's such a great
> idea, do it yourself.
 
 martin, i'm going to take a leaf from the great rusty russell's book,
 because i was very impressed with the professional way in which he
 dealt with someone who posted such immature and out-of-line comments:
 he rewrote them in a much more non-hostile manner and then replied to
 that.

 so, here goes: i'm copying the above few [relevant] paragraphs
 below, then rewriting them, here:

> > 
> >  ... and you are tied in to the decisions made by the linux kernel
> >  developers.
> 
> Yes, this is very true: we are all somewhat at the mercy of their
> decisions.  However, fortunately, they had the foresight to work
> with free software, so any of us can try something different, if
> we wish.
>
> i am slightly confused by your message, however: forgive me for
> asking this but you are not expecting us to implement such a radical
> redesign, are you?

 martin, hi, thank you for responding.

 well... actually, as it turns out, the l4linux and l4ka people have
 already done most of the work!!

 i believe you may have missed part of my message (it was a bit long, i
 admit) and i thank you for the opportunity, that your message presents,
 to reiterate this: l4linux _exists_ - last time i checked (some months
 ago) it had a port of 2.6.11 to the L4 microkernel.

 so, in more ways than one, no i am of course not expecting people to
 just take orders from someone as mad as myself :)

 i really should reiterate this: i _invite_ people to _consider_ the
 direction that processor designs - not just any "off-the-wall"
 processor designs but _mainstream_ x86-compatible processor designs -
 are likely to take.  and they are becoming more and more parallel.

 the kinds of questions that the experienced linux kernel
 maintainers and developers really need to ask is: can the
 present linux kernel design _cope_ with such parallelism?

 is there an easier way?

 that's mainly why i wished you "good luck" :)

 l.

 p.s. martin.  _don't_ do that again.  i don't care who you are:
 internet archives are forever and your rudeness will be noted
 by google-users and other search-users - long after you are dead.

Re: what's next for the linux kernel?

[Rik van Riel]

On Mon, 3 Oct 2005, Luke Kenneth Casson Leighton wrote:

>  well... actually, as it turns out, the l4linux and l4ka people have
>  already done most of the work!!

And I am sure they have reasons for not submitting their
changes to the linux-kernel mailing list.  They probably
know something we (including you) don't know.

Switching out the low level infrastructure does NOT help
with scalability.  The only way to make the kernel more
parallelizable is by changing the high level code, ie.
Linux itself.

Adding a microkernel under Linux is not going to help
with anything you mentioned so far.

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Chase Venters]

I'd venture to say that Linux scalability is fantastic. This also sounds like 
a repeat of a debate that happened ten years ago.

I too was intrigued by Andrew's comment about 'finishing the kernel', though 
I'm guessing (albeit without ever having spoken to Andrew personally) that it 
was partially in jest. What it does suggest, though, is a point that KDE 
desktop developer Aaron Seigo has made recently about the focus moving up the 
stack.

If we are admirably tackling the problems of hardware compatibility, 
stability, scalability and we've implemented most of the important features 
that belong in the kernel, then a lot of the development fire for a so-called 
complete Linux system is going to have to move up the stack - into the 
userland.

Indeed, adding 100 cores to my Pentium 4 isn't going to do me a damned bit of 
good when Akregator goes to query some 40 RSS feeds and Kontact blocks, 
refusing to process GUI events. It's also not going to make compiling a 
single .c file any faster. 

I have no doubt that the bright minds here on LKML will continue to find 
places to improve Linux's scalability, but that certainly doesn't require 
rebuilding the kernel - we're already doing remarkably well in the 
scalability department.

The bottom line is that the application developers need to start being clever 
with threads. I think I remember some interesting rumors about Perl 6, for 
example, including 'autothreading' support - the idea that your optimizer 
could be smart enough to identify certain work that can go parallel.

As dual cores and HT become more popular, the onus is going to be on the 
applications, not the OS, to speed up. 

Regards,
Chase Venters

On Sunday 02 October 2005 08:10 pm, Luke Kenneth Casson Leighton wrote:
> ... words ...

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 08:27:45PM -0500, Chase Venters wrote:

> The bottom line is that the application developers need to start being clever 
> with threads. 

 yep!  ah.  but.  see this:

 http://lists.samba.org/archive/samba-technical/2004-December/038300.html

 and think what would happen if glibc had hardware-support for
 semaphores and mutexes.

> I think I remember some interesting rumors about Perl 6, for 
> example, including 'autothreading' support - the idea that your optimizer 
> could be smart enough to identify certain work that can go parallel.

 http://www.ics.ele.tue.nl/~sander/publications.php
 http://portal.acm.org/citation.cfm?id=582068
 http://csdl.computer.org/comp/proceedings/acsd/2003/1887/00/18870237.pdf

 to get the above references, put in "holland parallel code
 analysis tools" into google.com.

 put in "parallel code analysis tools" into google.com for a different
 set.

 l.

Re: what's next for the linux kernel?

[Tushar Adeshara]

Hi all,
I am equally interested to see how all this will affect embedded world.
While processors will have to go parallel, I agree, there will be more
then one processor (may be uniprocessor) embedded in devices like
cellphone, MP3 player etc. against one on our desk and in server
rooms. Linux has to work on this devices too.


--
Regards,
Tushar
--------------------
It's not a problem, it's an opportunity for improvement. Lets improve.

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:
 > On Sun, Oct 02, 2005 at 08:27:45PM -0500, Chase Venters wrote:
 > 
 > > The bottom line is that the application developers need to start being clever 
 > > with threads. 
 > 
 >  yep!  ah.  but.  see this:
 > 
 >  http://lists.samba.org/archive/samba-technical/2004-December/038300.html
 > 
 >  and think what would happen if glibc had hardware-support for
 >  semaphores and mutexes.

Let me guess... nothing? Overhead of locking depends on data-structures
used by application/library and their access patterns: one thread has to
wait for another to finish with the shared resource. Implementing
locking in hardware is going to change nothing here (barring really
stupid implementations of locking primitives). Especially as we are
talking about blocking primitives, like pthread semaphore or mutex: an
entry into the scheduler will by far outweigh any advantages of
raw-metal synchronization.

 > 
 > > I think I remember some interesting rumors about Perl 6, for 
 > > example, including 'autothreading' support - the idea that your optimizer 
 > > could be smart enough to identify certain work that can go parallel.

Fortran people automatically parallelize loops for a _long_ time.

 > 
 >  http://www.ics.ele.tue.nl/~sander/publications.php
 >  http://portal.acm.org/citation.cfm?id=582068
 >  http://csdl.computer.org/comp/proceedings/acsd/2003/1887/00/18870237.pdf
 > 
 >  to get the above references, put in "holland parallel code
 >  analysis tools" into google.com.

PS: I wonder why Luke Kenneth Casson Leighton, Esq., while failing to
spell the Grandeur of his Appellative with the full Capitalization in
not a single From header humble readers of this Thread have a rare Honor
to witness, insists on referring to his interlocutors in minuscule only?

Does this correlate with an abnormally frequent usage of word
"condescending" in this discussion?

Nikita.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 07:04:35PM +0400, Nikita Danilov wrote:

 dude.

 chill.

 out.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 07:04:35PM +0400, Nikita Danilov wrote:
> Luke Kenneth Casson Leighton writes:
>  > On Sun, Oct 02, 2005 at 08:27:45PM -0500, Chase Venters wrote:
>  > 
>  > > The bottom line is that the application developers need to start being clever 
>  > > with threads. 
>  > 
>  >  yep!  ah.  but.  see this:
>  > 
>  >  http://lists.samba.org/archive/samba-technical/2004-December/038300.html
>  > 
>  >  and think what would happen if glibc had hardware-support for
>  >  semaphores and mutexes.
> 
> Let me guess... nothing? 

 interesting.

> Overhead of locking depends on data-structures
> used by application/library and their access patterns: one thread has to
> wait for another to finish with the shared resource.

 yes.

> locking in hardware is going to change nothing here (barring really
> stupid implementations of locking primitives). Especially as we are
> talking about blocking primitives, like pthread semaphore or mutex: an
> entry into the scheduler will by far outweigh any advantages of
> raw-metal synchronization.
 
 so what would, in your opinion, be a good optimisation?
 
 the references i found (just below) are to tool chains or research
 projects for code or linker-level analysis and parallelisation tools.

 what would, in your opinion, be a good way for hardware to assist
 thread optimisation, at this level (glibc)?

 assuming that you have an intelligent programmer (or some really good
 and working parallelisation tools) who really knows his threads?



>  >  http://www.ics.ele.tue.nl/~sander/publications.php
>  >  http://portal.acm.org/citation.cfm?id=582068
>  >  http://csdl.computer.org/comp/proceedings/acsd/2003/1887/00/18870237.pdf
>  > 
>  >  to get the above references, put in "holland parallel code
>  >  analysis tools" into google.com.
> 
> PS: I wonder why Luke Kenneth Casson Leighton, Esq., while failing to

 can i invite you to consider, when replying to these lists, to consider
 instead of treating it as a location where you can piss over anyone
 that you do not believe to be in any way your equal or in fact the equal
 of anyone, to instead consider the following template for your replies:

 	okay, right.
	* i do/don't get what this guy is saying.
	* i do/don't have an alternative idea (here it is / sorry)
	* here's what's wrong / right with what he's saying.
	* here's where it can/can't be done better.

 the bits that are missing from your reply are:
 
 * "you do/don't get where i'm going with this"
 * you haven't specified an alternative idea
 * you've outlined what's wrong but not what's right
 * you haven't specified how it can be done better.

 i therefore conclude that you are bully.  a snob.

 i _really_ detest bullying - and that's what you are doing.

 intellectual bullying.

 stop it.

 so i sent some messages saying "i think the kernel developers could be
 wrong in their design strategy" so FRIGGIN what?

 prove me right or prove me wrong.

 or shut up, or add my email address to your killfile.

 _don't_ be an intellectual snob.

 l.

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:

[...]

 > 
 >  assuming that you have an intelligent programmer (or some really good
 >  and working parallelisation tools) who really knows his threads?

Well, I'd like to have a hardware with CAS-n operation for one
thing. But what would this buy us? Having different kernel algorithms
for x86 and mythical cas-n-able hardware is not viable.

[...]

 > 
 >  can i invite you to consider, when replying to these lists, to consider
 >  instead of treating it as a location where you can piss over anyone
 >  that you do not believe to be in any way your equal or in fact the equal
 >  of anyone, 

As this self-contradictory (unless equivalence is not reflective in your
world) description is not the way I treat "these lists", I'll --instead
of pointing out to errors in your proposal-- follow (with improvements)
your advice, and re-interpret it the way I want:

 > 	* i do/don't get what this guy is saying.

Indeed. But maybe this is at least partially because "this guy" fails to
abide to the elementary rules of grammar?

One who really wants to communicate with other people, follows common
conventions that exist specifically on purpose of making communication
possible. If you want to engage into a discussion with other people, you
should put aside considerations of your own convenience or habit, and
stick to the common protocol. It's that simple.

[...]

 > 
 >  so i sent some messages saying "i think the kernel developers could be
 >  wrong in their design strategy" so FRIGGIN what?

Pray, do tell me you typed this without a shift key. I'd very much do
like to not destroy the very essence of 20 years of accumulated
experience.

 > 
 >  or shut up, or add my email address to your killfile.

Please do this with my address in exchange: suddenly, I am
overwhelmed with a glimmering presentiment of sharing this location with
a lot of worthy people.

Nikita.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 09:15:57PM +0400, Nikita Danilov wrote:
> Luke Kenneth Casson Leighton writes:
> 
> [...]
> 
>  > 
>  >  assuming that you have an intelligent programmer (or some really good
>  >  and working parallelisation tools) who really knows his threads?
> 
> Well, I'd like to have a hardware with CAS-n operation for one
> thing. 

 CAS - compare and swap - by CAS-n i presume that you mean effectively a
 SIMD CAS instruction?

> But what would this buy us? 

 you do not say :)  i am genuinely interested to hear what it would buy.

> Having different kernel algorithms
> for x86 and mythical cas-n-able hardware is not viable.

 if i can get an NPTL .deb package for glibc for x86 only it would tend
 to imply that that isn't a valid conclusion: am i missing something?

 cheers,

 l.
 
 

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:
 > On Tue, Oct 04, 2005 at 09:15:57PM +0400, Nikita Danilov wrote:
 > > Luke Kenneth Casson Leighton writes:
 > > 
 > > [...]
 > > 
 > >  > 
 > >  >  assuming that you have an intelligent programmer (or some really good
 > >  >  and working parallelisation tools) who really knows his threads?
 > > 
 > > Well, I'd like to have a hardware with CAS-n operation for one
 > > thing. 
 > 
 >  CAS - compare and swap - by CAS-n i presume that you mean effectively a
 >  SIMD CAS instruction?

An instruction that atomically compares and swaps n independent memory
locations with n given values. cas-1 (traditional compare-and-swap) is
enough to implement lock-less queue, cas-2 is enough to implement
double-linked lists, and was used by Synthesis lock-free kernel
(http://citeseer.ist.psu.edu/massalin91lockfree.html).

To be precise, cas-1 is theoretically enough to implement double-linked
lists too, but resulting algorithms are not pretty at all.

 > 
 > > But what would this buy us? 
 > 
 >  you do not say :)  i am genuinely interested to hear what it would buy.

Nothing. That was an instance of "rhetorical question", sorry that I
made not this clear enough.

 > 
 > > Having different kernel algorithms
 > > for x86 and mythical cas-n-able hardware is not viable.
 > 
 >  if i can get an NPTL .deb package for glibc for x86 only it would tend
 >  to imply that that isn't a valid conclusion: am i missing something?

Yes: this is Linux _Kernel_ mailing list, and I was talking about kernel
code and kernel algorithms.

 > 
 >  cheers,
 > 
 >  l.
 >  

Nikita.

 >  

Re: what's next for the linux kernel?

[Rik van Riel]

On Tue, 4 Oct 2005, Luke Kenneth Casson Leighton wrote:

>  	okay, right.
> 	* i do/don't get what this guy is saying.
> 	* i do/don't have an alternative idea (here it is / sorry)
> 	* here's what's wrong / right with what he's saying.
> 	* here's where it can/can't be done better.

It would help if you added something from the third and
fourth bullet points to your posts, instead of sticking
with just the first two.

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

[everyone but rik & alan hit delete physically or mentally _now_ ...]

rik, alan, i replied privately seeking your input on a post i was
developing in reply to your earlier comments - the ones before this
thread went mad even by my standards.

in amongst the pigeons, there's one pigeon that goes "meow":
i figured you might have missed it: it's got the words
"PLEASE REVIEW" as part of the subject.

sorry to everyone else for having to use the list as a private
communications channel.

l.

Re: what's next for the linux kernel?

[David Weinehall]

On Thu, Oct 06, 2005 at 01:07:44AM +0100, Luke Kenneth Casson Leighton wrote:
[snip]
> in amongst the pigeons, there's one pigeon that goes "meow":
> i figured you might have missed it: it's got the words
> "PLEASE REVIEW" as part of the subject.

Ohhhh, I think your shift key suddenly started to work...

[snip]


Regards: David Weinehall
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: what's next for the linux kernel?

[Rik van Riel]

On Thu, 6 Oct 2005, Luke Kenneth Casson Leighton wrote:

> rik, alan, i replied privately seeking your input on a post i was
> developing in reply to your earlier comments 

> "PLEASE REVIEW" as part of the subject.

I've got better things to do than saving you from yourself.

I replied to your original email in an attempt to educate
some of the other readers on the linux-kernel mailing list.
Private email does not have this same benefit, so there
was no reason for me to reply.

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 01:23:18PM -0400, Rik van Riel wrote:
> On Thu, 6 Oct 2005, Luke Kenneth Casson Leighton wrote:
> 
> > rik, alan, i replied privately seeking your input on a post i was
> > developing in reply to your earlier comments 
> 
> > "PLEASE REVIEW" as part of the subject.
> 
> I've got better things to do than saving you from yourself.
> 
> I replied to your original email in an attempt to educate
> some of the other readers on the linux-kernel mailing list.
> Private email does not have this same benefit, so there
> was no reason for me to reply.
 
 oh, right.  okay.

 i am disappointed by your response.
 
 i will therefore, instead of soliciting your input _before_
 sending to the LKLM, such that the quality of information sent
 to the list is much higher, now need to go ahead without the
 benefit of your advice, such that your input will be required
 as a "fait accomplit" in order to educate other LKML readers,
 with the unfortunate side-effect of an increased amount
 of noise.

 should you change your mind on this matter i would be honoured
 to receive review comments prior to responding, which will
 be in the next few days.

 l.





-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

http://www.eet.com/in_focus/embedded_systems/OEG20021213S0029

	It was decided at the beginning that we would design
	a system-on-chip (SoC) platform, which yields the
	best unit price when manufactured in high volume. The
	usual approach would be to license all the technology
	from third party suppliers, [...]  [but] we didn't
	want to deal with huge NRE and royalty fees. Also,
	we would not get the necessary know-how that is often
	a determining factor when designing a new product in
	today's ever decreasing time-to-market.

	So, we decided to follow the Linux open source
	philosophy and build our first platform on open
	source technology. We took several open source IPs
	from OpenCores [http://www.opencores.org] and integrate
	them into an underlying hardware SoC platform optimized
	for running Linux.

	[...]

		... As the main processor we chose OpenRISC 1200
	[http://www.opencores.org/pnews.cgi/list/or1k?no_loop=yes],
	a 32-bit RISC processor that comes with a stable GNU
	Toolchain and a port of small footprint Microcontroller
	Linux, the uClinux.

	The next critical part of the whole project was to set up a
	scheme on how to connect all the IPs in a modular way so that
	we could configure the platform for different embedded
	applications.         [...]

	We found out that a central configurable block
	interconnecting the processor and peripheral IPs did
	the trick.  [...]  we created a tool that automatically
	generated this central block  [...] and automatically
	configures Linux kernel and device drivers for that
	particular application.

	As embedded developers often find out, it is difficult
	to start writing and testing software, if hardware
	designers are still designing the hardware. It is
	necessary to parallel these two tasks in order to meet
	             ^^^^^^^^^^^^^^^^^^^^^^^^
	today's critical time-to-market schedules. In addition,
	each group can provide some test cases to the other
	as we found out.

these people designed the _entire_ embedded system - from free software
licensed components.

the processor design.

the software toolchain.

the kernel running on the free software licensed processor design.


it CAN be done.  it HAS been done.

convincing yourselves that you "must have hardware before you will get
off your fat asses" is _so_ self-disempowering.  STOP IT.

you - the linux kernel designers - are an extremely powerful
group who quite literally could hold the technical world to
ransom if you so chose (albeit for a very brief amount of time until
someone considered your actions to be the equivalent of a
"bus-running-over" event).

god help the world when you decide to actually say "thank you
for your hardware.  next time, consult us on what should be
in it _before_ you finalise its design".

l.

Re: what's next for the linux kernel?

[Al Viro]

On Fri, Oct 07, 2005 at 01:38:41AM +0100, Luke Kenneth Casson Leighton wrote:
> self-disempowering

*plonk*

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 08:22:20PM +0100, Luke Kenneth Casson Leighton wrote:
> On Thu, Oct 06, 2005 at 01:23:18PM -0400, Rik van Riel wrote:
> > On Thu, 6 Oct 2005, Luke Kenneth Casson Leighton wrote:
> > 
> > > rik, alan, i replied privately seeking your input on a post i was
> > > developing in reply to your earlier comments 
> > 
> > > "PLEASE REVIEW" as part of the subject.
> > 
> > I've got better things to do than saving you from yourself.
> > 
> > I replied to your original email in an attempt to educate
> > some of the other readers on the linux-kernel mailing list.

 my first reaction to your message was one of disappointment: my initial
 reply was therefore rather formal and curt, and i did not think much
 about this bit of your message.

 after some thought, i wanted to make it clear that i would
 welcome such education just as much as anyone else would.

 l.

Re: what's next for the linux kernel?

[Joe Bob Spamtest]

Luke Kenneth Casson Leighton wrote:
>  p.s. martin.  _don't_ do that again.  i don't care who you are:
>  internet archives are forever and your rudeness will be noted
>  by google-users and other search-users - long after you are dead.

and who are you, the thought police? Get off your high horse.

I'm sure he's well aware of the consequences of posting to this list, as 
I'm sure we all are. Hell, even *I* know all my mails to this list are 
going to be archived for eternity.

Look, if you want to be a productive member of our community, stop 
bitching about the way things *should* be, and submit some patches like 
everyone else. Code talks. Bullshit ... well, it doesn't do much but sit 
around stinking the place up.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 12:36:20PM -0700, Joe Bob Spamtest wrote:
> The point being: If and when the industry switches its focus to highly 
> parallel systems, Linux will shortly follow. 

 joe: hi, thanks for responding.  i believe this to be a very
 sound strategy, and given the technical expertise of the kernel
 developers i have confidence in their abilities to pull that off.

 personally i find that i like a bit of a run-up and/or advance notice
 of major paradigm shifts.  on the basis that other people might also
 want to know, i initiated this discussion yesterday and it seems like
 forever already! :)

 l.

 oh, and joe?  my wife is the one with the high horse, not me.
 she qualified for the national british dressage championships which
 was last month, and came 17th in the country, at elementary
 level, on her beautiful pony, blue.  i am very proud of her.

 http://www.bdchampionships.co.uk

Re: what's next for the linux kernel?

[Christoph Hellwig]

Let's hope these posts will stop when the UK starts to allow serving
drinks after 23:00.  Post from half-drunk people that need to get a life
don't really help a lot.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Sun, Oct 02, 2005 at 11:49:57PM +0100, Christoph Hellwig wrote:
> Let's hope these posts will stop when the UK starts to allow serving
> drinks after 23:00.  Post from half-drunk people that need to get a life
> don't really help a lot.

 hi, christoph,

 i assume that your global world-wide distribution of this message
 was a mistake on your part.  but, seeing as it _has_ gone out to
 literally thousands of extremely busy people, i can only apologise
 to them on your behalf for the mistake of wasting their valuable
 time.

 let's also hope that people who believe that comments such as the one
 that you have made are useful and productive also think about the
 consequences of doing so, bear in mind that internet archives are
 forever, and also that they check whether the person that they are
 criticising drinks at _all_.

 personally, my average consumption of alcohol can be measured
 as approx 1 bottle per decade.  and i'm not talking meths.

 if you don't like what i have to say, and don't want to listen,
 even with a pinch of salt to me rambling, learn how to set
 up a killfile, and use it.  and think more before hitting the
 reply-to-all button.  key.  whatever.

 l.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Willy Tarreau]

On Mon, Oct 03, 2005 at 12:24:16AM +0100, Luke Kenneth Casson Leighton wrote:
> On Sun, Oct 02, 2005 at 11:49:57PM +0100, Christoph Hellwig wrote:
> > Let's hope these posts will stop when the UK starts to allow serving
> > drinks after 23:00.  Post from half-drunk people that need to get a life
> > don't really help a lot.
> 
>  hi, christoph,
> 
>  i assume that your global world-wide distribution of this message
>  was a mistake on your part.  but, seeing as it _has_ gone out to
>  literally thousands of extremely busy people, i can only apologise
>  to them on your behalf for the mistake of wasting their valuable
>  time.

If you think this, you don't know Christoph then. We all know him
for his "warm words" and his frankness. Sometimes he may be quite
a bit excessive, but here I tend to agree with him. He simply meant
that these long threads are often totally useless and consume a lot
of time to real developers. Generally speaking, calling developers
to tell them "hey, you work the wrong way" is not productive. If you
tell them that you can improve their work and show some code, it's
often more appreciated. Yes I know you provided some links to sites,
but without proposing one real guideline nor project. You'd better
have posted something like "announce: merging of l4linux into
mainline" and telling people that you will start a slow, non-intrusive
merging work of one of your favorite project. You'd have got lots
of opposition but this would have been more productive than just
speaking about philosophy.

>  let's also hope that people who believe that comments such as the one
>  that you have made are useful and productive also think about the
>  consequences of doing so, bear in mind that internet archives are
>  forever, and also that they check whether the person that they are
>  criticising drinks at _all_.

[ cut all the uninteresting info about your drinking habits that you
  sent to the whole world and which will be archived forever ]

Regards,
Willy

Re: what's next for the linux kernel?

[Kurt Wall]

On Sun, Oct 02, 2005 at 11:49:57PM +0100, Christoph Hellwig took 8 lines to write:
> Let's hope these posts will stop when the UK starts to allow serving
> drinks after 23:00.  Post from half-drunk people that need to get a life
> don't really help a lot.

As posts from fully drunk people will help, either, albeit they might be
more entertaining.

Kurt
-- 
Violence is the last refuge of the incompetent.
		-- Salvor Hardin

Re: what's next for the linux kernel?

[Kurt Wall]

[...]

> with me so far? :)

Yes, and getting more annoyed at your condescending tone with each
paragraph.
 
> and, what is the linux kernel?
> 
> it's a daft, monolithic design that is suitable and faster on
> single-processor systems, and that design is going to look _really_
> outdated, really soon.

Andrew Tannenbaum said the same thing in the early 1990s. That we're
here still having this discussion >10 years later is telling. Dr.
Tannenbaum might have been acadmeically and theoretically correct,
but, with a nod to OS X, the Linux kernel has proven itself by
implementation and has proven to be remarkably adaptable.

Check back in ten years. Or just come back when you've completed
implementing all the beauteous features you're selling.

> in short, basically, if you follow and agree with the logic, the
> linux kernel - as maintained by linus - is far from complete.
> 
> i therefore invite you to consider the following strategy:

And the e.e. cummings affectation is even *more* annoying than your
condescension.

Kurt
-- 
Blood flows down one leg and up the other.

Re: what's next for the linux kernel?

[David Leimbach]

> > it's a daft, monolithic design that is suitable and faster on
> > single-processor systems, and that design is going to look _really_
> > outdated, really soon.
>
> Andrew Tannenbaum said the same thing in the early 1990s. That we're
> here still having this discussion >10 years later is telling. Dr.
> Tannenbaum might have been acadmeically and theoretically correct,
> but, with a nod to OS X, the Linux kernel has proven itself by
> implementation and has proven to be remarkably adaptable.

Why are you nodding to OS X?  It's not a real micokernel either.  It
just happens to have all the foobage of a microkernel in a rather
monolithic design.  The reason that the bsd personality is in the same
address space as the mach bits is because they didn't want to deal
with the overheads of the message passing from kernel to userspace.

The L4 people figured out how to get a lot of those inefficiencies to
disappear and L4Linux is quite "performant".  In some cases, L4Linux
can be used to provide a device driver for other L4 threads that would
normally have to write their own [in user space and even with
respectable performance
http://www.ertos.nicta.com.au/Research/ULDD/Performance.pml]

That's an interesting re-use and combination of several philosophies
if you ask me.

There is a lot of "what's next for linux" going on behind the scenes
and the current path of linux is apparently good enough for
accomplishing it.

- Dave

Re: what's next for the linux kernel?

[Nick Piggin]

Allow me to apply Rusty's technique, if you will.

Luke Kenneth Casson Leighton wrote:
> Hi,
> 
 > Can all you great kernel hackers, who only know a little bit
 > less than me and have only built a slightly less successful
 > kernel than I have, stop what you are doing and do it my way
 > instead?
 >

Hi Luke,

Thanks for your concise and non-rambling letter that is actually
readable - a true rarity on lkml these days.

To answer your question: I think we would all be happy to examine
your ideas when you can provide some real numbers and comparisions
and actual technical arguments as to why they are better than the
current scheme we have in Linux.

   Nick

PS. I am disappointed not to have seen any references to XML in
your proposal. May I suggest you adopt some kind of XML format
for your message protocol?

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: what's next for the linux kernel?

[Jon Masters]

On 10/2/05, Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:

> as i love to put my oar in where it's unlikely that people
> will listen, and as i have little to gain or lose by doing
> so, i figured you can decide for yourselves whether to be
> selectively deaf or not:

Hi Luke,

Haven't seen you since I believe you gave a somewhat interesting talk
on FUSE at an OxLUG a year or more back. I don't think anyone here is
selectively deaf, but some might just ignore you for such comments :-)

> what prompted me to send this message now was a recent report
> where linus' no1 patcher is believed to be close to overload,
> and in that report, i think it was andrew morton, it was
> said that he believed the linux kernel development rate to be
> slowing down, because it is nearing completion.

There was some general bollocks about Andrew being burned out, but
that wasn't what the point was as far as I saw it - more about how
things could be better streamlined than a sudden panic moment.

> i think it safe to say that a project only nears completion
> when it fulfils its requirements and, given that i believe that
> there is going to be a critical shift in the requirements, it
> logically follows that the linux kernel should not be believed
> to be nearing completion.

Whoever said it was?

> with me so far? :)

I don't think anyone with moderate grasp of the English language won't
have understood what you wrote above. They might not understand why
you said it, but that's another issue.

> the basic premise: 90 nanometres is basically... well...
> price/performance-wise, it's hit a brick wall at about 2.5Ghz, and
> both intel and amd know it: they just haven't told anyone.

But you /know/ this because you're a microprocessor designer as well
as a contributor to the FUSE project?

> anyone (big) else has a _really_ hard time getting above 2Ghz,
> because the amount of pipelining required is just... insane
> (see recent ibm powerpc5 see slashdot - what speed does it do?
> surprise: 2.1Ghz when everyone was hoping it would be 2.4-2.5ghz).

I think there are many possible reasons for that and I doubt slashdot
will reveal any of those reasons. The main issue (as I understand it)
is that SMT/SMP is taking off for many applications and manufacturers
want to cater for them while reducing heat output - so they care less
about MHz than about potential real world performance.

> so, what's the solution?

> well.... it's to back to parallel processing techniques, of course.

Yes. Wow! Of course! Whoda thunk it? I mean, parallel processing!
Let's get that right into the kern...oh wait, didn't Alan and a bunch
of others already do that years ago? Then again, we might have missed
all of the stuff which went into 2.2, 2.4 and then 2.6?

> well - intel is pushing "hyperthreading".

Wow! Really? I seem to have missed /all/ of those annoying ads. But
please tell me some more about it!

> and, what is the linux kernel?

> it's a daft, monolithic design that is suitable and faster on
> single-processor systems, and that design is going to look _really_
> outdated, really soon.

Why? I happen to think Microkernels are really sexy in a Computer
Science masturbatory kind of way, but Linux seems to do the job just
fine in real life. Do we need to have this whole
Microkernel/Monolithic conversation simply because you misunderstood
something about the kind of performance now possible in 2.6 kernels as
compared with adding a whole pointless level of message passing
underneath?

Jon.

Re: what's next for the linux kernel?

[Miklos Szeredi]

> But you /know/ this because you're a microprocessor designer as well
> as a contributor to the FUSE project?

AFAIK Luke never contributed to the FUSE project.  Hopefully that
answers your question.

FUSE and microkernels are sometimes mentioned together, but I believe
there's a very important philosophical difference:

FUSE was created to ease the development and use of a very _special_
group of filesystems.  It was never meant to replace (and never will)
the fantastically efficient and flexible internal filesystem
interfaces in Linux and other monolithic kernels.

On the other hand, the microkernel approach is to restrict _all_
filesystems to the more secure, but less efficient and less flexible
interface.  Which is stupid IMO.

Miklos

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 06:00:56PM +0200, Miklos Szeredi wrote:
> > But you /know/ this because you're a microprocessor designer as well
> > as a contributor to the FUSE project?
> 
> AFAIK Luke never contributed to the FUSE project.  Hopefully that
> answers your question.
 
 wrong.

 i added xattr support to fuse, for use in selinux.  it's a long story.

	 http://www.ussg.iu.edu/hypermail/linux/kernel/0409.2/1441.html

 and yes, for the record, i am just as comfortable with hardware
 designs as with software, having designed a massively parallel
 encryption algorithm capable of doing up to about 16384-bit
 block sizes with key sizes of up to around 8192-bits, (which
 unfortunately wasn't very fast in software - you can't have
 everything), came up with some significant improvements to
 the plessey/imperial-uni/man-uni ALICE parallel transputer
 network as a third year project, and also provided aspex,
 the massively-parallel SIMD processor company, with enough
 new material and ideas in four months for them to have to
 register six new patents.

 ... why are you people bothering to attempt to go "oh, this
 guy must not know anything therefore we'll waste the list's
 time with our opinions on whether he cannot do anything",
 such that i have to refute you, and look like a complete
 egg-head jumped-up i'm-better-than-you horn-blowing tosser?

 stop it!

 everyone has their level and areas of expertise: instead of
 turning this into a pissing contest, be glad and humbled for
 an opportunity to learn from each other.

 l.

Re: what's next for the linux kernel?

[Miklos Szeredi]

> > > But you /know/ this because you're a microprocessor designer as well
> > > as a contributor to the FUSE project?
> > 
> > AFAIK Luke never contributed to the FUSE project.  Hopefully that
> > answers your question.
>  
>  wrong.
> 
>  i added xattr support to fuse, for use in selinux.  it's a long story.
> 
> 	 http://www.ussg.iu.edu/hypermail/linux/kernel/0409.2/1441.html
> 

Well, adding is a different thing from contribution.

Contribution is when you add something and also share it with the
maintainer of said software.

I can't remember you doing that, so that's why I said what I said.

Miklos

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Mon, Oct 03, 2005 at 03:20:46PM +0100, Jon Masters wrote:
> On 10/2/05, Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> 
> > as i love to put my oar in where it's unlikely that people
> > will listen, and as i have little to gain or lose by doing
> > so, i figured you can decide for yourselves whether to be
> > selectively deaf or not:
> 
> Hi Luke,

 hellooo jon.

> Haven't seen you since I believe you gave a somewhat interesting talk
> on FUSE at an OxLUG a year or more back. 

 good grief, that long ago?

> I don't think anyone here is
> selectively deaf, but some might just ignore you for such comments :-)
 
 pardon?  oh - yes, i'm counting on it, for a good signal/noise
 ratio.  sad to recount, the strategy ain't workin too well,
 oh well.  i've received about 10 hate mails so far, i _must_
 be doing _something_ right.


> > i think it safe to say that a project only nears completion
> > when it fulfils its requirements and, given that i believe that
> > there is going to be a critical shift in the requirements, it
> > logically follows that the linux kernel should not be believed
> > to be nearing completion.
> 
> Whoever said it was?
 
 istrc it was in andrew morton's interview / comments :)

> > the basic premise: 90 nanometres is basically... well...
> > price/performance-wise, it's hit a brick wall at about 2.5Ghz, and
> > both intel and amd know it: they just haven't told anyone.
> 
> But you /know/ this because you're a microprocessor designer as well
> as a contributor to the FUSE project?
 
 i have been speaking on a regular basis with someone who
 has been dealing for nearly twenty years now with processor
 designs (from a business perspective, for assessing high-tech
 companies for investment and recruitment purposes).  i have
 been fortunate enough to have the benefit of their experience
 in assessing the viability of chip designs.

 i haven't created silicon, but i've studied processor designs until
 they were coming out of my ears.

 ... you are mistaken on one point, though: my work on fuse proved
 unsuccessful because i was a) running out of time b) running out of
 reasons to continue (the deal fell through) c) i ran into an error
 message in selinux: the "please try later" one, which flummoxed me but
 i now believe to be due to a crash in the userspace stuff.  maybe.
 urk.  it's been a year.

 anyway, we digress.



 

> > anyone (big) else has a _really_ hard time getting above 2Ghz,
> > because the amount of pipelining required is just... insane
> > (see recent ibm powerpc5 see slashdot - what speed does it do?
> > surprise: 2.1Ghz when everyone was hoping it would be 2.4-2.5ghz).
> 
> I think there are many possible reasons for that and I doubt slashdot
> will reveal any of those reasons. 

 probably :)

> The main issue (as I understand it)
> is that SMT/SMP is taking off for many applications and manufacturers
> want to cater for them while reducing heat output - so they care less
> about MHz than about potential real world performance.

 pipelining.  pipelining.  latency between blocks.

 halving the microns should quadruple the speed: the distance is halved
 so light has half the distance to travel and ... darn, can't remember
 the other reason for the other factor-of-two.

 if the latency between sub-blocks is large (or becomes
 relevant), then it doesn't matter _what_ microns you attempt to
 run in, your design will asymtote towards an upper speed limit.

 if you're having to pipeline down to the level of 2-bit adders with 16
 stages in order to do 32-bit adds at oh say 4ghz, you _know_ you're in
 trouble.


> > so, what's the solution?
> 
> > well.... it's to back to parallel processing techniques, of course.
> 
> Yes. Wow! Of course! Whoda thunk it? I mean, parallel processing!
> Let's get that right into the kern...oh wait, didn't Alan and a bunch
> of others already do that years ago? Then again, we might have missed
> all of the stuff which went into 2.2, 2.4 and then 2.6?

 jon, jon *sigh* :)  i meant _hardware_ parallel processing - i wasn't
 referring to anything led or initiated by the linux kernel, but instead
 to the simple conclusion that if hardware is running out of steam in
 uniprocessor (monster-pipelined; awful-prediction;
 let's-put-five-separately-designed-algorithms-for-divide-into-the-chip-and-take-the-answer-of-the-first-unit-that-replies sort of design)
 then chip designers are forced to parallelise.
 
> > well - intel is pushing "hyperthreading".
> 
> Wow! Really? I seem to have missed /all/ of those annoying ads. But
> please tell me some more about it!
 
 make me.  nyer :)

> > and, what is the linux kernel?
> 
> > it's a daft, monolithic design that is suitable and faster on
> > single-processor systems, and that design is going to look _really_
> > outdated, really soon.
> 
> Why? I happen to think Microkernels are really sexy in a Computer
> Science masturbatory kind of way, but Linux seems to do the job just
> fine in real life. Do we need to have this whole
> Microkernel/Monolithic conversation simply because you misunderstood
> something about the kind of performance now possible in 2.6 kernels as
> compared with adding a whole pointless level of message passing
> underneath?

 i'll answer rik's point later when i've thought about it some
 more: in short, rik has concluded that because i advocated
 message passing that somehow his SMP improvement work
 (isolating data structures) is irrelevant - far from it: such
 improvements would prove, i believe, to be a significant additional
 augmentation, unburdening both a monolithic _and_ a microkernel'd linux
 kernel from some of the cru-joze nastiness of SMP.

 if there are better ways, _great_.

 love to hear them.

 l.

Re: what's next for the linux kernel?

[Jon Masters]

On Mon, Oct 03, 2005 at 09:22:39PM +0100, Luke Kenneth Casson Leighton wrote:

> On Mon, Oct 03, 2005 at 03:20:46PM +0100, Jon Masters wrote:

> > On 10/2/05, Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:

>  hellooo jon.
> 
> > Haven't seen you since I believe you gave a somewhat interesting talk
> > on FUSE at an OxLUG a year or more back. 
> 
>  good grief, that long ago?

Indeed. Time flies.

> > I don't think anyone here is
> > selectively deaf, but some might just ignore you for such comments :-)
>  
>  pardon?  oh - yes, i'm counting on it, for a good signal/noise
>  ratio.  sad to recount, the strategy ain't workin too well,
>  oh well.  i've received about 10 hate mails so far, i _must_
>  be doing _something_ right.

I hate to sound like "one of them" but nothing you've said is revolutionary -
really it's not - but you're saying it as if you just dreamed it up today and
/that's/ what got people annoyed. One of those pseudo-intellectual Starbucks
moments, if you will.

> > > the basic premise: 90 nanometres is basically... well...
> > > price/performance-wise, it's hit a brick wall at about 2.5Ghz, and
> > > both intel and amd know it: they just haven't told anyone.
> > 
> > But you /know/ this because you're a microprocessor designer as well
> > as a contributor to the FUSE project?
>  
>  i have been speaking on a regular basis with someone who
>  has been dealing for nearly twenty years now with processor
>  designs (from a business perspective, for assessing high-tech
>  companies for investment and recruitment purposes).  i have
>  been fortunate enough to have the benefit of their experience
>  in assessing the viability of chip designs.

In other words, no. I'm not a processor designer either (very few people
are) but I do have a lot of experience with Xilinx FPGAs and I'll add
something of relevence to the end of this message. The point really is
that some people here know a great deal more than you do about this (and
I'm not saying I'm one of them), they're going to rightfully feel a
little annoyed if you start preeching to the choir.

> > The main issue (as I understand it)
> > is that SMT/SMP is taking off for many applications and manufacturers
> > want to cater for them while reducing heat output - so they care less
> > about MHz than about potential real world performance.
> 
>  pipelining.  pipelining.  latency between blocks.

Would you mind learning to use capitali[sz]ation in your mail? It's
really not very easy to read what you write. Was the above intended to
be an actual sentence (in which case I can't see any clauses in the
above) or just random words which - if said together - will summon some
mystical power upon us?

> > > so, what's the solution?
> > 
> > > well.... it's to back to parallel processing techniques, of course.
> > 
> > Yes. Wow! Of course! Whoda thunk it? I mean, parallel processing!
> > Let's get that right into the kern...oh wait, didn't Alan and a bunch
> > of others already do that years ago? Then again, we might have missed
> > all of the stuff which went into 2.2, 2.4 and then 2.6?
> 
>  jon, jon *sigh* :)

My point was that some very much more cleaver people have worked on this
for a very long time already. Alan did a lot of cool stuff in the
beginning, what Ingo does now just scares me, etc.

> i meant _hardware_ parallel processing

Old hat. It's been worked on for over a decade and some of it is working
out now in the form of concept processors that mix FPGAs with CPUs.

> - i wasn't referring to anything led or initiated by the linux kernel,
> but instead to the simple conclusion that if hardware is running out of
> steam in uniprocessor (monster-pipelined; awful-prediction;
>  let's-put-five-separately-designed-algorithms-for-divide-into-the-chip-and-take-the-answer-of-the-first-unit-that-replies sort of design)
>  then chip designers are forced to parallelise.

So simple in fact that it's already done in most common hardware. Why
else would we have any offload chips at all?

Please help me to understand the value of your original message. I've
read it a few times, but it continues to allude me.

Jon.

Re: what's next for the linux kernel?

[Jason Stubbs]

Luke Kenneth Casson Leighton wrote:
>  halving the microns should quadruple the speed: the distance is halved
>  so light has half the distance to travel and ... darn, can't remember
>  the other reason for the other factor-of-two.

2 dimensions?

--
Jason Stubbs

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 10:33:16AM +0900, Jason Stubbs wrote:
> Luke Kenneth Casson Leighton wrote:
> > halving the microns should quadruple the speed: the distance is halved
> > so light has half the distance to travel and ... darn, can't remember
> > the other reason for the other factor-of-two.
> 
> 2 dimensions?
 
 Voltage-squared.  capacitance.  when you go down the microns, your
 capacitance drops and the voltage squared goes down, too.

 0.65nm is 1.2v

 0.45 is aiming for 0.9 volts.
 
 silicon germanium is going to hit a limit real soon.
 you can't go below 0.8 volts, that's the gate "off" threshold.

 l.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Marc Perkel]

I think it's time for some innovative thinking and for people to step 
outside the Linux box and look around at other operating systems for 
some good ideas. I'll run through a few ideas here.

Reiser 4 - The idea of building a file system on top of a database is 
the right way to go. Reiser is onto something here and this is a 
technology that needs to be built upon. It's current condition is a 
little on the week side - no ACLs for example - but the underlying 
concept is ound.

Novell Netware type permissions. ACLs are a step in the right direction 
but Linux isn't any where near where Novell was back in 1990. Linux lets 
you - for example - to delete files that you have no read or write 
access rights to. Netware on the other hand prevents you from deleting 
files that you can't write to and if you have no right it is as if the 
file isn't there. You can't even see it in the directory. Netware also 
has inherited permissions like Windows and Samba has and this is doing 
it right. File systems and individual directories should be able to be 
flagged as casesensitive/insensitive. Permissions need to be fine 
grained and easy to use. Netware is a good example to emulate.

The bootup sequence of Linux is pathetic. What an ungodly mess. The 
FSTAB file needs to go and a smarter system needs to be developed. I 
know this isn't entirely a kernel issue but it is somewhat related.

I think development needs to be done to make the kernel cleaner and 
smarter rather than just bigger and faster. It's time to look at what 
users need and try to make Linux somewhat more windows like in being 
able to smartly recover from problems. Perhaps better error messages 
that your traditional kernel panic or hex dump screen of death.

The big challenge for Linux is to be able to put it in the hands of 
people who don't want to dedicate their entire life to understanding all 
the little quirks that we have become used to. The slogan should be 
"this just works" and is intuitive.

Anyhow - before I piss off too many people who are religiously attached 
to Linux worshiping - I'll quit not. ;)

Marc Perkel
Linux Visionary

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 12:47:25PM -0700, Marc Perkel wrote:

> The bootup sequence of Linux is pathetic. What an ungodly mess. The 
> FSTAB file needs to go and a smarter system needs to be developed. I 
> know this isn't entirely a kernel issue but it is somewhat related.
 
 depinit.  written by richard lightman.  easily located with google.

 on relatively inexpensive amd 2100 hardware, depinit results
 in a startup time to console login of 5 seconds, and x-windows
 in a further 3.


 this is probably as good a time as any to mention this:

 depinit on a 2.6 kernel has had to have a small script added which does
 a sleep 3; kill -HUP <itself> - i.e. "kill -HUP 1".

 if this is not done, then any child program that sends a signal to
 process 1 is NOT SEEN.

 richard believes the problem to be actually in the 2.6 kernel.

 whilst /sbin/init only catches one signal, depinit catches quite
 literally _all_ of them.

 i'm relaying this from memory, so some of the above may be inaccurate.


> I think development needs to be done to make the kernel cleaner and 
> smarter rather than just bigger and faster. 

 actually, on embedded systems the linux 2.6 kernel is bigger
 and slower, which has prompted a large number of embedded systems
 designers to stick with the [by now abandoned] 2.4 series.

> Marc Perkel
> Linux Visionary
  ^^^^^ ^^^^^^^^^
 wha-heeeeey !

 
 my main concern, btw, is that by the time linux kernel developers
 "receive hardware to play with", it's already too late.

 the hardware decisions have already been made.

 you - worthy as you are and the work you are doing is -
 are treated as second class citizens by the companies
 manufacturing hardware.

 time to put the horse before the cart.

 l.

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Chase Venters]

On Tuesday 04 October 2005 02:47 pm, Marc Perkel wrote:
> The bootup sequence of Linux is pathetic. What an ungodly mess. The
> FSTAB file needs to go and a smarter system needs to be developed. I
> know this isn't entirely a kernel issue but it is somewhat related.
>
> I think development needs to be done to make the kernel cleaner and
> smarter rather than just bigger and faster. It's time to look at what
> users need and try to make Linux somewhat more windows like in being
> able to smartly recover from problems. Perhaps better error messages
> that your traditional kernel panic or hex dump screen of death.
>
> The big challenge for Linux is to be able to put it in the hands of
> people who don't want to dedicate their entire life to understanding all
> the little quirks that we have become used to. The slogan should be
> "this just works" and is intuitive.

I agree with the basic sentiment here.

fstab is pretty traditional - and you're right, it really isn't a kernel thing 
per se. Let's not forget the value of simplicity though. fstab works and does 
its job well. I do think it would be interesting for a distribution to 
experiment with different approaches, though -- something with the emerging 
hardware abstraction layer perhaps.

As for error messages... the equivalent of the Linux kernel panic is basically 
the Windows BSOD. Neither one of them should appear in the day to day use of 
the system as they indicate bugs. Linux is actually the clear winner here, I 
think, because a Windows BSOD gives you a single hex code and no indication 
of what happened, except for very vague codes like 
"PAGE_FAULT_IN_NON_PAGED_AREA". I'd much rather have a backtrace :) In any 
case, I'm watching the work on kdump with a keen interest.

Really, I think the whole issue of usability isn't tied directly to the 
kernel. The kernel has been making leaps and bounds in making this easy for 
userspace to deal with (where the approaches to solve the problem belong). 
Sysfs is an obvious great example.

Work on dbus and HAL should give us good improvements in these areas. One 
remaining challenge I see is system configuration - each daemon tends to 
adopt its own syntax for configuration, which means that providing a GUI for 
novice users to manage these systems means attacking each problem separately 
and in full. Now I certainly wouldn't advocate a Windows-style registry, 
because I think it's full of obvious problems. Nevertheless, it would be nice 
to have some kind of configuration editor abstraction library that had some 
sort of syntax definition database to allow for some interesting work on 
GUIs.

In any case, I think pretty much all of this work lives outside the kernel. 
There is one side note I'd make about booting - my own boot process has to 
wait forever for my Adaptec SCSI controller to wake up. It would be 
interesting if bootup initialization tasks could be organized into dependency 
levels and run in parallel, though as I'm a beginner to the workings of the 
kernel I'm not entirely sure how possible this would be.

Cheers,
Chase Venters

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

On Tue, 04 Oct 2005 18:40:33 CDT, Chase Venters said:
> Work on dbus and HAL should give us good improvements in these areas. One
> remaining challenge I see is system configuration - each daemon tends to
> adopt its own syntax for configuration, which means that providing a GUI for
> novice users to manage these systems means attacking each problem separately 
> and in full. Now I certainly wouldn't advocate a Windows-style registry,
> because I think it's full of obvious problems. Nevertheless, it would be nice
> to have some kind of configuration editor abstraction library that had some
> sort of syntax definition database to allow for some interesting work on
> GUIs.

Anybody who tries to do this without at least understanding the design choices
made by AIX's SMIT tool deserves to re-invent it, poorly.

> In any case, I think pretty much all of this work lives outside the kernel.

Amen to that - although the whole hotplug/udev/sysfs aggregation has at least
made a semi-sane way to find out from userspace what the kernel thinks is going on...

Are there any drivers out there that don't play nice with sysfs? If so, should
a mention of them be added to http://kerneljanitors.org/TODO ?


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 01:35:57AM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 04 Oct 2005 18:40:33 CDT, Chase Venters said:
> > Work on dbus and HAL should give us good improvements in these areas. One

 HAL is great.  dbus should have been shot at birth and the people who
 initiated it without looking at freedce should have been fired.

 l.

Re: what's next for the linux kernel?

[Steven Rostedt]

On Tue, 4 Oct 2005, Chase Venters wrote:

> As for error messages... the equivalent of the Linux kernel panic is basically
> the Windows BSOD. Neither one of them should appear in the day to day use of
> the system as they indicate bugs. Linux is actually the clear winner here, I
> think, because a Windows BSOD gives you a single hex code and no indication
> of what happened, except for very vague codes like
> "PAGE_FAULT_IN_NON_PAGED_AREA". I'd much rather have a backtrace :) In any
> case, I'm watching the work on kdump with a keen interest.
>

And what about kexec?  To be able to boot into another kernel on a kernel
bug and still have access to all the memory and the system state of the
bug.  That's pretty cool.  It would be like Windows going straight to
Safe-Mode on a BSOFD without a reboot.

> In any case, I think pretty much all of this work lives outside the kernel.
> There is one side note I'd make about booting - my own boot process has to
> wait forever for my Adaptec SCSI controller to wake up. It would be
> interesting if bootup initialization tasks could be organized into dependency
> levels and run in parallel, though as I'm a beginner to the workings of the
> kernel I'm not entirely sure how possible this would be.
>

I've been thinking of at least trying to see what would happen if I
threaded the do_initcalls in main.c but lately I haven't had the time.

-- Steve

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

> > In any case, I think pretty much all of this work lives outside the kernel.
> > There is one side note I'd make about booting - my own boot process has to
> > wait forever for my Adaptec SCSI controller to wake up. It would be
> > interesting if bootup initialization tasks could be organized into dependency
> > levels and run in parallel, though as I'm a beginner to the workings of the
> > kernel I'm not entirely sure how possible this would be.

 again - depinit.

 the only thing that richard lightman didn't spend time on was the
 splitting out of hotplug / hotplug scripts such that depinit could be
 told "okay the ethernet's up now, all eth0 dependencies can now proceed"
 or "okay the scsi controller is up now, all fstab entries depending
 on this controller can now proceed".

 richard's example scripts split out "var", "usr", "home", "boot" as
 separate dependencies (actually they're symlinks to the "mount"
 pseudo-dependency) on which things like pretty much all services 
 that need to do syslogging depend on the "var" dependency you get the
 idea.

 so as long as it's hotplug that gets the notifications, great.

 if it's _not_ hotplug that's receiving the notifications, such that
 device driver initialisation cannot be delayed because you're looking at
 calling some boot-rom initialisation stuff, then, sorry, nope - you're
 gonna have to just wait for that SCSI controller to get its act
 together :)

 but again, this is off-topic for the original discussion and is again
 userspace not kernel oh well, in for a penny in for a pound.

 l.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Tue, Oct 04, 2005 at 06:40:33PM -0500, Chase Venters wrote:

> Work on dbus and HAL should give us good improvements in these areas. One 

 dbus.  total waste of several man-years of effort that could have been
 better spent in e.g. removing the dependency of posix draft 4 threads
 from freedce (which i finally did last month) and e.g. adding an ultra-fast
 shared-memory transport plugin.

 hal.  _excellent_.  look forward to it being ported to win32 so that
 it's useful for the kde-win32 port etc. etc.

> remaining challenge I see is system configuration - each daemon tends to 
> adopt its own syntax for configuration, which means that providing a GUI for 
> novice users to manage these systems means attacking each problem separately 
> and in full. 

 that's quite straightforward to deal with but it _does_ mean using a
 unified approach to writing APIs.

 NT solved this problem by writing graphical tools in c and then
 adopting dce/rpc as the means to administer the services both locally
 _and_ remotely.

 wholesale.  utterly.  everything.  right from the simplest things like
 rebooting the machine, through checking the MAC addresses of cards
 (NetTransportEnum) all the way up to DNS administration - yes,
 dnsadmin.exe will write out DNS zone files in proper bind format.

 it's quite a brave choice to take.

> Now I certainly wouldn't advocate a Windows-style registry, 
> because I think it's full of obvious problems. 

 such as? :)

 they're not obvious to me.  at the risk of in-for-penny, in-for-pound
 _radically_ off-topic discussion encouragement here, and also for
 completeness should someone come back to the archives in some years or
 months and go "what obvious problems", could you kindly elaborate?

 one i can think of is "eek, my system's broken, eek, i can't even use
 vi to edit the configs".

 and having described the problem, then.. .. well... actually...
 it's simply dealt with:

 http://www.bindview.com/Services/RAZOR/Utilities/Unix_Linux/ntreg_readme.cfm

 todd sabin wrote a linux filesystem driver which is read-only, so at
 least half the work's done.

 (and the reactos people have written a complete implementation
 of a registry, btw).

> Nevertheless, it would be nice 
> to have some kind of configuration editor abstraction library that had some 
> sort of syntax definition database to allow for some interesting work on 
> GUIs.
 
 i have to say this: it's almost too radical, dude :)

 he he.


> In any case, I think pretty much all of this work lives outside the kernel. 

 ACK!

 ... well... not entirely.

 a "registry" - god help us - would need to be stored on a filesystem.
 and then, ideally, made accessible a la todd sabin's ntreg driver - via
 a POSIX interface (because the linux kernel doesn't _do_ anything other
 than POSIX filesystems *sigh*).  and that makes it also convenient to
 access from kernelspace, too.

 hey, you know what?  if linux got a registry, it would be possible for
 the kernel to access - and store, and communicate - persistent
 information.

 conveniently.

 hurrah.

 

Re: what's next for the linux kernel?

[Diego Calleja]

El Wed, 5 Oct 2005 11:26:50 +0100,
Luke Kenneth Casson Leighton <lkcl@lkcl.net> escribió:

> > Now I certainly wouldn't advocate a Windows-style registry, 
> > because I think it's full of obvious problems. 
> 
>  such as? :)


The ugly implementation (inside the kernel and as a big file instead of doing it as a
userspace in top of NTFS files which would have helped them to avoid lots of problems
that they were forced to solve in XP/2003, it's clear from their docs that they didn't
expected that registry could grow _so_ much), the fact that they use it to store at
the same time userspace configuration and internal kernel structures. The "idea" is
nice but the way they've implemented and used it is horrible - just take a look
at how they're using XML configuration files for IIS now... (I've been said that ISS will
detect when you're editing those configuration files and will reload them to duplicate
the changes you just made in the registry ..... ugh)


>  hey, you know what?  if linux got a registry, it would be possible for
>  the kernel to access - and store, and communicate - persistent
>  information

right - why you would want to do such thing is another story

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 01:04:10PM +0200, Diego Calleja wrote:

> El Wed, 5 Oct 2005 11:26:50 +0100,
> Luke Kenneth Casson Leighton <lkcl@lkcl.net> escribi?:
> 
> > > Now I certainly wouldn't advocate a Windows-style registry, 
> > > because I think it's full of obvious problems. 
> > 
> >  such as? :)
> 
> 
> The ugly implementation (inside the kernel and as a big file instead of doing it as a

 the nt 3.51 implementation got it right: userspace service (MSRPC
 service) with LPC (this is NT, based on Mach, so they have LPC which is
 message-passing - joy) communicating from userspace to kernelspace
 where necessary.

 nooo, it not okay to have registry in kernel.  _access_ to it (via
 ioctl's) yes.  _in_ kernel, friggin'ell'no.

 regarding the other points: yes, there's a per-user hive key, which is
 "overlaid" onto parts of the sub-tree.

 and yes, the previous poster is absolutely right: the benefits cannot
 be felt unless evvverrryyy service under the sun is also using it.

 ... but heck - we do configuration of pretty much every major service
 under the sun out of ldap, don't we?
 
 and openldap itself just got the ability to read its own
 config out of its own database, right?

 it's not _that_ far off, not _that_ unachievable, s/ldap/registry.

 there's just a few core services missing - initscripts is a good
 example - which nobody's yet had the nerve to tackle (afaik) and dump
 into LDAP.

 in that example, mostly because there's not much point unless
 you're also going to do something decent like put in proper
 dependencies (see depinit).

 anyway.  how on _earth_ did we get here, and please could
 someone advise me - and everyone else - of a more suitable
 location to discuss these matters?

 l.

Re: what's next for the linux kernel?

[Chase Venters]

On Wednesday 05 October 2005 05:26 am, Luke Kenneth Casson Leighton wrote:
> > Now I certainly wouldn't advocate a Windows-style registry,
> > because I think it's full of obvious problems.
>
>  such as? :)

If such a thing were even going to be attempted on UNIX, it would have to be 
so different than the NT approach that it would stop looking like a registry 
altogether.

For one thing, you need the ability to chroot / have multiple namespaces. It's 
totally possible that I as a user decide to install 78 different versions of 
Apache for some wild reason, and I expect my configuration settings to stay 
separate, damnit! 

Also, it would need to be rock-solid. Losing blocks on the disk should not 
mean losing the ability to boot the OS. Powering off in the middle of a write 
should not be fatal. 

What about applications attached to removable media? Should these applications 
assume that it is correct to store state in my registry? What happens when I 
then try to carry the media to another computer? If I wanted my settings to 
migrate, the application would need to be smart enough not to use the 
registry implementation, which makes it that much more worthless.

Why implement such cruft in the kernel? A user-space implementation is 
perfectly reasonable. An example that comes to mind is gconf, which uses XML 
files in a hierarchy to achieve something that looks sort of like a registry. 
Indeed, the requirements I briefly touch on above (which are just examples of 
the number of considerations I'm sure there really are) all point to an 
implementation we already have - a filesystem. Why reinvent the wheel?

Moreover, I think the idea of a system-wide registry is a bad idea altogether. 
In environments like GNOME or KDE it's generally OK because your applications 
are designed for the environment. Not so in the general Linux environment - 
we have tons and tons of applications we're capable of running that don't 
have the first clue about registries or why they would want it. So we're 
stuck either implementing it in all these applications (effectively forking 
them from the versions that used to work on every other UNIX), or they don't 
use the registry, in which case we've just added cruft that we don't even use 
(at the expense of confusing our end-users, well, to no end), or we choose to 
abandon these applications (not going to happen, ever).

KISS comes into play here, I think. A system registry is an interesting 
attempt to solve the universal configuration problem, but I think it does not 
adapt to the UNIX "way of thinking" well at all.

Regards,
Chase Venters

Re: what's next for the linux kernel?

[Michael Concannon]

Chase Venters wrote:

>On Wednesday 05 October 2005 05:26 am, Luke Kenneth Casson Leighton wrote:
>  
>
>>>Now I certainly wouldn't advocate a Windows-style registry,
>>>because I think it's full of obvious problems.
>>>      
>>>
>> such as? :)
>>    
>>
>
>If such a thing were even going to be attempted on UNIX, it would have to be 
>so different than the NT approach that it would stop looking like a registry 
>altogether.
>  
>
All good points, but perhaps the most compelling to me is that virtually 
every successful windows virus out there does its real damage by 
modifying the registry to replace key actions, associate bad actions 
with good ones and just generally screw the system up...

One could argue that this is no different than a hapless victim running 
as root getting his/her /etc/* files modified but:
a. the decentralization there makes it easy to distinguish those files 
which have been touched and how to fix them
b. they are all ASCII
c. they are not modified often, most almost never after initial system 
config
d. you don't have every app that runs mod'ing those files... (in fact 
few are even allowed to)
e. what the !@#$ would I want to cache my most recently visited URLs in 
the same place I decide where the entry hooks to my video driver live?

Anyone suggesting that Linux or Unix in general should inherit this, 
what I consider, most fatal flaw of all the flaws of windows should be 
dealt with harshly...

Sorry, could not resist responding - I cannot count the hours I have 
spent searching and clearing registry entries in family and friend's 
computers after they downloaded the latest cool virus...

/mike

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 11:10:55AM -0400, Michael Concannon wrote:

> All good points, but perhaps the most compelling to me is that virtually 
> every successful windows virus out there does its real damage by 
> modifying the registry to replace key actions, associate bad actions 
> with good ones and just generally screw the system up...
 
 the damage is done because "admin" rights are forced out of the control
 of the users and sysadmins and into the hands of the dumb-ass app
 writers, for both the setup stage and then the actual day-to-day
 usage of the app!

 the registry on NT has ACLs - which are completely irrelevant as far as
 users running as admin are concerned (because the dumb-ass app writers
 force them to).

 the nt registry - imagine it to be .... _like_ a filesystem, or _like_
 an LDAP server.

 except with proper ACLs and access controls [which everyone bypasses
 because duh it's windows duh, not because it's impossible to do a decent
 job with the API and its implementation].

 l.

Re: what's next for the linux kernel?

[Michael Concannon]

Luke Kenneth Casson Leighton wrote:

>On Thu, Oct 06, 2005 at 11:10:55AM -0400, Michael Concannon wrote:
>
>  
>
>>All good points, but perhaps the most compelling to me is that virtually 
>>every successful windows virus out there does its real damage by 
>>modifying the registry to replace key actions, associate bad actions 
>>with good ones and just generally screw the system up...
>>    
>>
> 
> the damage is done because "admin" rights are forced out of the control
> of the users and sysadmins and into the hands of the dumb-ass app
> writers, for both the setup stage and then the actual day-to-day
> usage of the app!
>
> the registry on NT has ACLs - which are completely irrelevant as far as
> users running as admin are concerned (because the dumb-ass app writers
> force them to).
>
> the nt registry - imagine it to be .... _like_ a filesystem, or _like_
> an LDAP server.
>
> except with proper ACLs and access controls [which everyone bypasses
> because duh it's windows duh, not because it's impossible to do a decent
> job with the API and its implementation].
>  
>
No question that one could limit the damage with various tweaks to 
permissions and access controls, but it is the very centralization of 
information with such vastly disparate purposes (into a single file) 
that is the flaw here...

You can view it, think about it and talk about it as a "file-system" and 
that is fine..  much like /proc or sysfs, but when the system crashes:

1. It _is_ a file: registry.dat
2. It is a binary file at that...
3. That file has become a dumping ground for everything that every app 
thinks is "important" and of course every app writer thinks everything 
they write is the most important thing ever - I am sure a have never 
done such a thing :-)

I guess you could argue that #3 is the fault of the app writers and not 
the architecture, but clearly the current state is the  result of those 
app writers traveling the path of least resistance, so viewed as a whole 
the architecture is to blame regardless...  While it may be wrong for 
people to steal money left on a table out in front of the bank, the bank 
should have  known that this would result and put the money inside...

#2 is an issue because of the complexity of the system which must be 
function to perform the most basic functions of system recovery... 

If you can boot a floppy/pendrive/cd and mount it with vi then it is a 
disk in need of service...

If you cannot, it is a brick in need of re-installation...

I have booted linux a number of times with an NT drive as a slave and 
recovered it.   I have not ever done the inverse...

I hate vi with a passion, more often than not, I have to hit :q! a few 
times before I remember what I have to type, but the fact is, it works 
when nothing else does and it has saved a lot of systems for me...

#2 is also an issue of security because with very simple and reliable 
tools, one can track and monitor changes to key files, one can impose 
any level of security with any level of granularity (perhaps too many 
grains with SELinux, but that is your choice).  Before there was 
tripwire, there were lots of people who wrote basically the same thing 
in plain simple shell/perl scripts and it worked...

#2 is also an issue of backup and restoration...  If it is a 
file-system, it does not provide any useful methods of incremental 
backup and restoration...

There is no equivalent of:
cd etc/xinet.d ; cvs update -A

/etc _is_ a filesystem with all the benefits that come with it...

/tmp is also a great file-system and a much better place to cache all 
that "important" application specific temporary data...  If they want to 
save state, there is:

/etc/<appname>.conf for site-wide setups
~/.<appname> for user-specific state...

I was trying to stay out this thread - clearly I failed :-)  No value 
judgement intended for any of the comments made, this thread is a like a 
car accident on a busy highway...  everyone knows they are slowing 
things down by looking, but they cannot look away...

/mike

> l.
>
>  
>

Re: what's next for the linux kernel?

[Michael Concannon]

>
> I have booted linux a number of times with an NT drive as a slave and 
> recovered it.   I have not ever done the inverse...

ok, that rambled...

I meant that I did not need to do that with linux...  just used a 
floppy/cd/usb drive and edited files... 

With NT the only way I "recovered" was with a well timed backup of 
registry.dat or a binary image of the whole system...

Nothing incremental about it...

/mike

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 04:13:15PM -0400, Michael Concannon wrote:

> 1. It _is_ a file: registry.dat
> 2. It is a binary file at that...

 so don't make the implementation a file.

 or make the contents _of_ the file available via a file system interface.

 quoted for a second time in this thread, this link:

 http://www.bindview.com/Services/RAZOR/Utilities/Unix_Linux/ntreg_readme.cfm

 todd sabin's linux filesystem device driver, which understands nt
 registry fileformat.

	 NTREG
	 -----

	 This is a file system driver for linux, which
	 understands the NT registry file format. With it,
	 you can take registry files from NT, e.g., SAM,
	 SECURITY, etc., and mount them on linux. Currently,
	 it's read-only, though I may add read-write capability
	 in the future.

 http://www.bindview.com/Resources/RAZOR/Files/ntreg.tar.gz

 mentioned for a second time in this thread, the fact that reactos has
 read-write capability into nt hive key (registry) format.

 l.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 04:13:15PM -0400, Michael Concannon wrote:

> 1. It _is_ a file: registry.dat
> 2. It is a binary file at that...
> 3. That file has become a dumping ground for everything that every app 
> thinks is "important" and of course every app writer thinks everything 
> they write is the most important thing ever - I am sure a have never 
> done such a thing :-)
 
 s/"that file"/"openldap" and substitute "every app writer"
 for "every major free software developer we respect greatly which
 can store its data and/or configuration details in an LDAP database"
 and your evident distaste for "that file" looks a little like religious
 zealotry.

 i say that with the greatest respect.

 especially when "that file" is actually a database, just like
 Berkeley DB (and we all know and _love_ Berkeley DB).

 and especially in light it being possible to do a "decent" job, and
 make "that file" available via a POSIX filesystem interface.

 l.

> I guess you could argue that #3 is the fault of the app writers and not 
> the architecture, 

 yes.  i would say it's more to do with the dumb-ass nature of the app
 writers, yes.  typicall dumb-ass windows app writers give a shit about
 security and care greatly about making money hand-over-fist.
 
 whereas on linux it's far less likely for an app writer to
 be able to get away with a) making money b) friggin up security.  the
 distros wouldn't allow an app writer to get away with either.

 l.

Re: what's next for the linux kernel?

[Michael Concannon]

Luke Kenneth Casson Leighton wrote:

>On Thu, Oct 06, 2005 at 04:13:15PM -0400, Michael Concannon wrote:
>
>  
>
>>1. It _is_ a file: registry.dat
>>2. It is a binary file at that...
>>3. That file has become a dumping ground for everything that every app 
>>thinks is "important" and of course every app writer thinks everything 
>>they write is the most important thing ever - I am sure a have never 
>>done such a thing :-)
>>    
>>
> 
> s/"that file"/"openldap" and substitute "every app writer"
> for "every major free software developer we respect greatly which
> can store its data and/or configuration details in an LDAP database"
> and your evident distaste for "that file" looks a little like religious
> zealotry.
>  
>
I don't believe in religion :-) That is not to say I am atheist, just 
don't see the distinction between one mythology and the next...  but 
that really is another thread...

As for your prior comment regarding mounting the registry as a 
filesystem, I did see that and made a book mark for the next time I have 
to recover an NT box...  thanks :-)

However, I am now a little confused though (ok, I am always a little 
confused - see comment above on religion - now I am really confused).

If you concede that it need not be a binary file and it need not be 
centralized, whether by mounting a "filesystem" or not, then what are we 
talking about?  That is what we have with /etc /proc /sys?

Assuming you fix the issues of easy of editing it in "dead" filesystems, 
binary corruption, permissions and programming style of dumping crap in 
there, then I guess I could care less how that is implemented, proc is 
already a "virtual filesystem"....  

I think someone already mentioned that the issue is that the delta 
between an idealized NT registry (which has a few notable hurdles - see 
above) and what we have to day is simply a matter of "KISS".  What do 
you gain from complicating the system that cannot be gained with 
visualization tools on top of what is there? 

Someone wants XML in /proc?  Well, that's just fabulous they can write a 
virtual filesytem that accomplishes that on top what is there and leave 
the rest of us out of it :-)  If what they do becomes indespensible for 
a critical mass, then even better, we mount "xmlprocfs" in our future 
systems and are fat dumb and happy.

Back to your original point which seemed to be, at least to me, to try 
to re-evaluate the portioning problem between 
Hardware/Software/Drive/OS/User/Threads.  I agree, that the system 
appears to be strained and chaotic with all OSes chasing an ever 
increasing and impossibly large array of hardware and all-the while the 
future is even more complex as it seemingly must be a heavily 
parallelized future to compensate for the "end of Moore's law".  Given 
the hardware in question, though, I am not sure I that I see that Linux 
should go to micro-kernels to solve the problem...

<rant>
It seems to me that the driver for "correcting" this is actually closer 
to the hardware side...  I am flabbergasted as a hardware engineer that 
at this point in time with the time elapsed between today and the first 
PCs that things have evolved so little...

"drivers" should be the exception not the rule...

Few gadgets architecturally do anything different than anything else in 
that class of gadget to really _require_ a driver that could not be 
standardized.  Gadgets need user-space applications, but with all the 
well defined standards we have for talking to devices, there is no 
excuse with the wealth of compute power and storage that we have that we 
(the entire PC industry) are still shipping hacked, one-off drivers 
wither every new gadget...

The same applies to the x86 instruction set - waded through that beast 
(well all N volumes of it) recently?  WTF?  Its as if 199Million of 
those 200M gate chips are devoted to obfuscating the user interface....  
Most of those bits had a purpose at some point, but we don't seem to be 
converging to simpler interface...

I would fire me if ever I even proposed such a horrible design...  but 
then I don't design x86 processors...

If the CPUs and associated hardware started providing a more pleasant 
interface, then I am certain that Linux would respond by taking 
advantage of it...   We aren't there yet...

</rant>

/mike


> i say that with the greatest respect.
>
> especially when "that file" is actually a database, just like
> Berkeley DB (and we all know and _love_ Berkeley DB).
>
> and especially in light it being possible to do a "decent" job, and
> make "that file" available via a POSIX filesystem interface.
>
> l.
>
>  
>
>>I guess you could argue that #3 is the fault of the app writers and not 
>>the architecture, 
>>    
>>
>
> yes.  i would say it's more to do with the dumb-ass nature of the app
> writers, yes.  typicall dumb-ass windows app writers give a shit about
> security and care greatly about making money hand-over-fist.
> 
> whereas on linux it's far less likely for an app writer to
> be able to get away with a) making money b) friggin up security.  the
> distros wouldn't allow an app writer to get away with either.
>
> l.
>
>
>  
>

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Thu, Oct 06, 2005 at 05:53:40PM -0400, Michael Concannon wrote:

> I think someone already mentioned that the issue is that the delta 
> between an idealized NT registry (which has a few notable hurdles - see 
> above) and what we have to day is simply a matter of "KISS".  What do 
> you gain from complicating the system that cannot be gained with 
> visualization tools on top of what is there? 
 
 it is advocated that storing data in text format is easier
 to recover with a minimum of tools.

 if, as todd sabin's driver shows, you can present the binary data via a
 text-editable filesystem driver, then even _that_ argument goes away.

> Someone wants XML in /proc?  Well, that's just fabulous they can write a 
> virtual filesytem that accomplishes that on top what is there and leave 
> the rest of us out of it :-)  If what they do becomes indespensible for 
> a critical mass, then even better, we mount "xmlprocfs" in our future 
> systems and are fat dumb and happy.

 reiserfs already has had an XML plugin written for it, which resulted
 in the person it was written for thanking the author for writing the
 "best xml database they had ever seen".

 think structured storage and streams (which NTFS used to support, and
 which explorer.exe in nt 3.51 used to support before
 bill/someone-on-high ordered it to be taken out) and now apply that
 principle to XML instead.

> Back to your original point which seemed to be, at least to me, to try 
> to re-evaluate the portioning problem between 
> Hardware/Software/Drive/OS/User/Threads.  

 yes.  seems like ages ago.

> I agree, that the system 
> appears to be strained and chaotic with all OSes chasing an ever 
> increasing and impossibly large array of hardware and all-the while the 
> future is even more complex as it seemingly must be a heavily 
> parallelized future to compensate for the "end of Moore's law".  

 what - the constantly revised one, or the original one?

> Given 
> the hardware in question, though, I am not sure I that I see that Linux 
> should go to micro-kernels to solve the problem...

 i would be delighted to see #ifdef HAVE_L4_MICROKERNEL #endif in the
 linux source code.

 AS AN OPTION.

 that people could choose "bugger that damn l4 trash" or "bugger that
 monolithic trash" as they see fit.

 and the l4linux source code _is_ there for the linux kernel developers
 to pick up and incorporate.

 that they have not chosen to do so - even though it is GPL code -
 leaves me a little puzzled.

 
 
> <rant>
> It seems to me that the driver for "correcting" this is actually closer 
> to the hardware side...  I am flabbergasted as a hardware engineer that 
> at this point in time with the time elapsed between today and the first 
> PCs that things have evolved so little...
 
  ah _ha_.

  as a hardware engineer, what would _you_ like to see a
  modern parallel processor design have - that linux for that
  hardware would have an easy job of knocking the stuffing
  out of anything remotely attempting to come close to it?

  i.e. if you could have the proverbial cart before the
  proverbial horse, and could make decisions about the DESIGN
  of hardware - all of it - BEFORE it was dumped in your lap
  and you were basically impicitly told "we're giving you
  this for free and taking advantage of your willingness to go
  'cool hardware!  let's make it run linux".


  we've already had one suggestion: a CAS-n instruction (a SIMD
  compare-and-store) which can, according to the person who kindly
  suggested it, be used for managing doubly-linked lists.

  anyone got any more?

  want to send them to me, and i will summarise to those people that
  express an interest.

  [i _so_ want this thread to die].

> The same applies to the x86 instruction set - waded through that beast 
> (well all N volumes of it) recently?  WTF?  

 even _decoding_ the x86 instruction set, due to the massive
 number of exceptions / extensions, introduces significant
 instruction latency.
 
 a large amount of an x86 compatible processor is spent
 translating that crap into a simpler _internal_ microcode set
 of instructions.

 unberfrigginlievable.

 oh, these instructions are too slow.  _let's_ go an' add multimedia
 extensions.  *noooooooo.*

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Michael Concannon]

>><rant>
>>It seems to me that the driver for "correcting" this is actually closer 
>>to the hardware side...  I am flabbergasted as a hardware engineer that 
>>at this point in time with the time elapsed between today and the first 
>>PCs that things have evolved so little...
>>    
>>
> 
>  ah _ha_.
>
>  as a hardware engineer, what would _you_ like to see a
>  modern parallel processor design have - that linux for that
>  hardware would have an easy job of knocking the stuffing
>  out of anything remotely attempting to come close to it?
>
>  i.e. if you could have the proverbial cart before the
>  proverbial horse, and could make decisions about the DESIGN
>  of hardware - all of it - BEFORE it was dumped in your lap
>  and you were basically impicitly told "we're giving you
>  this for free and taking advantage of your willingness to go
>  'cool hardware!  let's make it run linux".
>  
>
Actually, no...  I'd like to see the OS equivalent of a GPU...  We know 
what the problems of an OS are now...  they have not changed for 25 
years...  and in those 25 years we have gathered an enormous library of 
lessons learned...

Let's learn from what we already know and take a step above push/pop/mov 
(ok a leap) and stop working around what is now basically 30 years of 
crap piled on the 8086 and create some hardware that is actual "Specific 
to the Application" rather than the other way around... 

/mike

ps. I am doing my part to kill this thread - "just say no"  ;-)

Re: what's next for the linux kernel?

[Michael Concannon]

>><rant>
>>It seems to me that the driver for "correcting" this is actually closer 
>>to the hardware side...  I am flabbergasted as a hardware engineer that 
>>at this point in time with the time elapsed between today and the first 
>>PCs that things have evolved so little...
>>    
>>
> 
>  ah _ha_.
>
>  as a hardware engineer, what would _you_ like to see a
>  modern parallel processor design have - that linux for that
>  hardware would have an easy job of knocking the stuffing
>  out of anything remotely attempting to come close to it?
>
>  i.e. if you could have the proverbial cart before the
>  proverbial horse, and could make decisions about the DESIGN
>  of hardware - all of it - BEFORE it was dumped in your lap
>  and you were basically impicitly told "we're giving you
>  this for free and taking advantage of your willingness to go
>  'cool hardware!  let's make it run linux".
>  
>
Actually, no...  I'd like to see the OS equivalent of a GPU...  We know 
what the problems of an OS are now...  they have not changed for 25 
years...  and in those 25 years we have gathered an enormous library of 
lessons learned...

Let's learn from what we already know and take a step above push/pop/mov 
(ok a leap) and stop working around what is now basically 30 years of 
crap piled on the 8086 and create some hardware that is actually 
"Specific to the Application" rather than the other way around... 

/mike

ps. I am doing my part to kill this thread - "just say no"  ;-)

Re: what's next for the linux kernel?

[Howard Chu]

Luke Kenneth Casson Leighton wrote:
>
>  ... but heck - we do configuration of pretty much every major service
>  under the sun out of ldap, don't we?
>
>  and openldap itself just got the ability to read its own
>  config out of its own database, right?
>
>  it's not _that_ far off, not _that_ unachievable, s/ldap/registry. 

I think both you and Michael have interesting points on this topic. Fyi, 
OpenLDAP now has dynamic config (accessible/modifiable via LDAP) but the 
backing store is still a bunch of flat files. There were two objectives 
here - (1) make every knob tunable via LDAP, and (2) don't prevent an 
admin from fixing things with vi if they have to. I've spent too many 
times rescuing systems in single-user mode with only /bin/sh, to ever 
commit to using a binary config database.

Yes, KISS is a good policy, you just have to understand what the 'It' is 
that you're talking about in each instance. Putting a filesystem driver 
on top of a registry.dat file seems to provide a simple user interface, 
so it *looks* like you're adhering to KISS, but the innards are still 
both complex and fragile. Hell, even the simplest filesystem driver you 
can write is a couple hundred lines of code.

The LDAP-enabled config engine in OpenLDAP looks more structured / more 
complex than the old flat slapd.conf file, but under the covers it's all 
still plain text. In one case, you're taking something very complex and 
putting a simple cover on it, in the other you have very simple building 
blocks and put a complex / richer interface on top of it. Guess which 
design is more likely to keep functioning in the face of a system failure.

The Unix programming philosophy is about taking small simple tools and 
combining them to perform more complex tasks. You could say it's one of 
the world's earliest object-oriented UIs. If you don't keep that in 
mind, and try to build complexity in starting at your most basic 
building blocks on the bottom (i.e., the kernel) then you're going to 
have a nightmare trying to keep anything you build on top of it working. 
One only need look at MS Windows to see how true this is.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Re: what's next for the linux kernel?

[Helge Hafting]

On Thu, Oct 06, 2005 at 10:20:01PM +0100, Luke Kenneth Casson Leighton wrote:
> On Thu, Oct 06, 2005 at 04:13:15PM -0400, Michael Concannon wrote:
> 
> > 1. It _is_ a file: registry.dat
> > 2. It is a binary file at that...
> > 3. That file has become a dumping ground for everything that every app 
> > thinks is "important" and of course every app writer thinks everything 
> > they write is the most important thing ever - I am sure a have never 
> > done such a thing :-)
>  
>  s/"that file"/"openldap" and substitute "every app writer"
>  for "every major free software developer we respect greatly which
>  can store its data and/or configuration details in an LDAP database"
>  and your evident distaste for "that file" looks a little like religious
>  zealotry.
> 
Well, there are many differences between the windows registry and 
openldap on unix.  First, openldap is voluntary.  There are plenty
of linux systems not using ldap at all, and still using whatever apps
they want. People deploy ldap when they think the benefits outweighs
the added complexity. :-)

Helge Hafting 

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

helge, yours is the kind of comment which is welcome on linux visionaries and
i wish, i wissshhh, for the thread on lkml with this subject to DIEEEE.

l.

On Sun, Oct 09, 2005 at 12:27:09AM +0200, Helge Hafting wrote:
> On Thu, Oct 06, 2005 at 10:20:01PM +0100, Luke Kenneth Casson Leighton wrote:
> > On Thu, Oct 06, 2005 at 04:13:15PM -0400, Michael Concannon wrote:
> > 
> > > 1. It _is_ a file: registry.dat
> > > 2. It is a binary file at that...
> > > 3. That file has become a dumping ground for everything that every app 
> > > thinks is "important" and of course every app writer thinks everything 
> > > they write is the most important thing ever - I am sure a have never 
> > > done such a thing :-)
> >  
> >  s/"that file"/"openldap" and substitute "every app writer"
> >  for "every major free software developer we respect greatly which
> >  can store its data and/or configuration details in an LDAP database"
> >  and your evident distaste for "that file" looks a little like religious
> >  zealotry.
> > 
> Well, there are many differences between the windows registry and 
> openldap on unix.  First, openldap is voluntary.  There are plenty
> of linux systems not using ldap at all, and still using whatever apps
> they want. People deploy ldap when they think the benefits outweighs
> the added complexity. :-)
> 
> Helge Hafting 
> 
> 

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: what's next for the linux kernel?

[Horst von Brand]

Marc Perkel <marc@perkel.com> wrote:
> I think it's time for some innovative thinking and for people to step
> outside the Linux box and look around at other operating systems for
> some good ideas. I'll run through a few ideas here.

Great! Where are your patches?

> Reiser 4 - The idea of building a file system on top of a database is
> the right way to go.

It's insanity. I'll give it to you in writing, if you want.

>                      Reiser is onto something here and this is a
> technology that needs to be built upon. It's current condition is a
> little on the week side - no ACLs for example - but the underlying
> concept is ound.

There are many who disagree. Even violently...

> Novell Netware type permissions. ACLs are a step in the right
> direction but Linux isn't any where near where Novell was back in
> 1990.

Details?

>       Linux lets you - for example - to delete files that you have no
> read or write access rights to.

Yep. This is spelled U-N-I-X. You just (re)discovered the most "discovered"
(by newbies) "security bug" in Unix.

>                                 Netware on the other hand prevents you
> from deleting files that you can't write to and if you have no right
> it is as if the file isn't there.

Sorry, but you don't "delete a file" here, you delete a /link/ to a file.
Yes, if it was the last one the file goes away, but that is just a special
case.

>                                   You can't even see it in the
> directory. Netware also has inherited permissions like Windows

Inherited permissions are /evil/, as you need to check all the way up to /
to find out if something is allowed or not.

>                                                                and
> Samba has and this is doing it right. File systems and individual
> directories should be able to be flagged as
> casesensitive/insensitive.

Here file names aren't made up of "letters" that have "cases", they are
just random strings of (almost unrestricted) bytes. So you are free to
interpret them as japanese or arabic. Your choice.

>                            Permissions need to be fine grained and
> easy to use.

"Fine grained" is almost exactly the opposite to "easy to use".

>              Netware is a good example to emulate.

Go ahead and write your filesystem then! It might even be of some use to be
able to read Netware volumes. I'm not trying to stop you.

> The bootup sequence of Linux is pathetic.

This has nothing to do with Linux (the kernel).

>                                           What an ungodly mess. The
> FSTAB file needs to go and a smarter system needs to be developed. I
> know this isn't entirely a kernel issue but it is somewhat related.

In some 30 years nobody has been able to come up with something better. If
you have a better idea, I'm all ears. Note that the mechanism would have to
be just as simple (or hopefully simpler).

> I think development needs to be done to make the kernel cleaner

Sign up with the kernel janitors.

>                                                                 and
> smarter

That's part of the idea around here.

>         rather than just bigger

Nobody is advocating "bigger". It grows mostly to handle new pieces of
hardware, or new requirements. Again, if you have concrete proposals on how
to shrink the kernel, everybody here will jump of joy. Big is slow, and
thus bad.

>                                 and faster.

What is bad about "faster"?

>                                             It's time to look at what
> users need

The distribution people are looking into that, don't worry.

>            and try to make Linux somewhat more windows like in being
> able to smartly recover from problems.

Please, everything /but/ "Windows style problem recovery"! I /much/ prefer
the system to crash than to (silently) paper over serious problems until it
is far too late to do anything about their effects.

>                                        Perhaps better error messages
> that your traditional kernel panic or hex dump screen of death.

Patches are more than welcome.

> The big challenge for Linux is to be able to put it in the hands of
> people who don't want to dedicate their entire life to understanding
> all the little quirks that we have become used to. The slogan should
> be "this just works" and is intuitive.

It is almost there, in the form of the more user-friendly distributions.
Again, help is higly appreciated there.

> Anyhow - before I piss off too many people who are religiously
> attached to Linux worshiping - I'll quit not. ;)

Thanks.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[D. Hazelton]

[-- Attachment #1.1: Type: text/plain, Size: 6885 bytes --]

On Tuesday 04 October 2005 19:47, Marc Perkel wrote:
> I think it's time for some innovative thinking and for people to
> step outside the Linux box and look around at other operating
> systems for some good ideas. I'll run through a few ideas here.
>
> Reiser 4 - The idea of building a file system on top of a database
> is the right way to go. Reiser is onto something here and this is a
> technology that needs to be built upon. It's current condition is a
> little on the week side - no ACLs for example - but the underlying
> concept is ound.

A filesystem built on top of a database? Isn't that introducing 
complexity into something that should be as simple as possible so 
that the number of possible errors is reduced?

But then, I do not have experience with filesystem design and 
implementation, so I cannot make a suggestion on this front.

> Novell Netware type permissions. ACLs are a step in the right
> direction but Linux isn't any where near where Novell was back in
> 1990. Linux lets you - for example - to delete files that you have
> no read or write access rights to. 

As someone else pointed out, this is because unlinking is related to 
your access permissions on the parent directory and not the file.

> Netware on the other hand 
> prevents you from deleting files that you can't write to and if you
> have no right it is as if the file isn't there. You can't even see 
> it in the directory. 


This is just adding a layer of security through hiding data. I 
personally don't like this idea for a number of reasons, not the 
least of which is that it is the access permissions of the directory 
that control whether or not you can see a file. This and the previous 
comment about unlinking of files is, IIRC, actually part of the POSIX 
standard.

> Netware also has inherited permissions like 
> Windows and Samba has and this is doing it right. 

This would only be a bonus with ACL's. It makes no real sense to 
propogate a directories permissions down to the files in a directory 
since an directory you can list the contents of has at least 1 
execute bit set.

> File systems and 
> individual directories should be able to be flagged as
> casesensitive/insensitive. 

This would be a rarely used feature and would break many tools. Having 
an extra bit would also require modifying the kernel and I doubt 
anyone wants to tackle such a job, as it would also break all extant 
filesystems.

> Permissions need to be fine grained and 
> easy to use. Netware is a good example to emulate.

I do agree with this, but have to point out that this is already 
allowed for under POSIX and a number of filesystems support this. 
It's called "POSIX ACL's" in the kernel configuration system. Since I 
don't use them on my home system (I see no need since it's just me 
and whatever hacker has managed to penetrate the system (to date: 0)) 
I do use ACL's (POSIX and otherwise) on all the systems I maintain 
for my various clients (providing the OS supports them)

> The bootup sequence of Linux is pathetic. What an ungodly mess. The
> FSTAB file needs to go and a smarter system needs to be developed.
> I know this isn't entirely a kernel issue but it is somewhat
> related.

I'll have to disagree about FSTAB - this is something that is at the 
peak of it's usefulness and changing or removing it would require the 
people that maintain the core utilities to rewrite mount(8) almost 
entirely.

However, when it comes to the boot sequence as controlled by init(8) I 
have to agree. I'm personally working on an entirely new set of 
init-scripts for my system and have thought about seeing if anyone 
has ever released an init(8) that is more functional than the basic 
GNU/FSF version. If there was an init(8) that could run the scripts 
in parallel I'd be using it as soon as I had a set of scripts lined 
up that were designed to be run in parallel.

> I think development needs to be done to make the kernel cleaner and
> smarter rather than just bigger and faster. It's time to look at
> what users need and try to make Linux somewhat more windows like in
> being able to smartly recover from problems. Perhaps better error
> messages that your traditional kernel panic or hex dump screen of
> death.

lol. Nope. The kernel panic could be refined to contain even more 
information and be even more user-friendly but it is definately 
light-years ahead of a Windows BSOD. Now if you're talking about the 
errors as seen by users of applications that's not a kernel issue and 
is the purview of the application developers.

> The big challenge for Linux is to be able to put it in the hands of
> people who don't want to dedicate their entire life to
> understanding all the little quirks that we have become used to.
> The slogan should be "this just works" and is intuitive.

Yep. I do agree with that. However, until the rest of the big 
companies catch up to the ones that already support Linux this will 
never happen. Several of my non-business maintenance clients have 
inquired abotu Linux and I've had to tell them to just stick with 
Windows because they rely on a rather large number of Windows only 
programs (that do _not_ run under Wine) and/or are not technically 
enough inclined to be able to handle the learning curve involved in 
moving to a non-MS operating system.  

The fact that I have gotten those inquiries means that the news about 
how stable Linux is is getting to the "mainstream" population. Only 
problem is that MS and the home-PC boom has landed the PC and the 
internet in the hands of too many people who are barely computer 
literate enough to use a mouse. (I'm speaking from experience. A 
large number of my private clients fit this description to a T. 
Although they are good as a continuing source of income :)  And with 
Windows being as "User Friendly" as it is, and with at least 75% of 
the major software firms still not supporting Linux, there is no way 
Linux can make any real inroads into the desktop market.

OTOH several of my clients have inquired about Linux because of it's 
security - it doesn't take a genius to see that the same reason 
Firefox made such a big dent in MS's hold on the browser market could 
work for Linux as well. And I have had more than one client ask if 
there was an alternative to Windows than ran well - I've always 
answered the same way: "Yes, but you would need to learn an all-new 
way of doing things." Every last one of them dropped it on hearing 
those words.

> Anyhow - before I piss off too many people who are religiously
> attached to Linux worshiping - I'll quit not. ;)

heh. keep it up. you've managed to turn a pointless argument of micro 
versus mono into something productive. (even if you didn't mean to)

DRH

[-- Attachment #1.2: 0xA6992F96300F159086FF28208F8280BB8B00C32A.asc --]
[-- Type: application/pgp-keys, Size: 1365 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: what's next for the linux kernel?

[Marc Perkel]

D. Hazelton wrote:

>  
>
>>Novell Netware type permissions. ACLs are a step in the right
>>direction but Linux isn't any where near where Novell was back in
>>1990. Linux lets you - for example - to delete files that you have
>>no read or write access rights to. 
>>    
>>
>
>As someone else pointed out, this is because unlinking is related to 
>your access permissions on the parent directory and not the file.
>
>  
>
Right - that's Unix "inside the box" thinking. The idea is to make the 
operating system smarter so that the user doesn't have to deal with 
what's computer friendly - but reather what makes sense to the user. 
 From a user's perspective if you have not rights to access a file then 
why should you be allowed to delete it?

Now - the idea is to create choice. If you need to emulate Unix nehavior 
for compatibility that's fine. But I would migrate away from that into a 
permissions paradygme that worked like Netware.

I started with Netware and I'm spoiled. They had it right 15 years ago 
and Linux isn't any where near what I was with Netware and DOS in 1990. 
Once you've had this kind of permission power Linux is a real big step down.

So - the thread is about the future so I say - time to fix Unix.

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Tue, 04 Oct 2005 22:49:03 PDT, Marc Perkel said:
>  From a user's perspective if you have not rights to access a file then 
> why should you be allowed to delete it?

Because it's your directory, dammit, and nobody else should be allowed to
clutter it with files you can't even read. :)

What's so hard to understand about that viewpoint?  Want to try to explain
the converse to a Windows/Netware user? "But it's *MY* folder, why can't I
get rid of this file I can't read and have no use for?"

Trying to make it "make sense to the user" without expecting the user to learn
at least a bit about what's going on is futile, as Alan Perlis understood:

When someone says "I want a programming language in which I need only
say what I wish done," give him a lollipop.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Nikita Danilov]

Marc Perkel writes:

[...]

 > Right - that's Unix "inside the box" thinking. The idea is to make the 
 > operating system smarter so that the user doesn't have to deal with 
 > what's computer friendly - but reather what makes sense to the user. 
 >  From a user's perspective if you have not rights to access a file then 
 > why should you be allowed to delete it?

Because in Unix a name is not an attribute of a file.

Files are objects that you read, write and truncate. They are
represented by inodes.

Separately from that, there is an indexing structure: directory
tree. Directories map symbolical names to inodes. Obviously, adding a
reference to an index, or removing it from one requires access
permission to the _index_ rather then to the object being referenced.

That two-level model of files and indexing on top of them is essential
to Unix due to the flexibility and conceptual economy it provides.

 > 
 > Now - the idea is to create choice. If you need to emulate Unix nehavior 
 > for compatibility that's fine. But I would migrate away from that into a 
 > permissions paradygme that worked like Netware.

And there are people believing that ITS (or VMS, or <insert your first
passion here>...) set the standard to follow. :-)

[...]

 > 
 > So - the thread is about the future so I say - time to fix Unix.

One thing is clear: it's too late to fix Netware. Why should Unix
emulate its lethal defects?

Nikita.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:

> Marc Perkel writes:
> 
> [...]
> 
>  > Right - that's Unix "inside the box" thinking. The idea is to make the 
>  > operating system smarter so that the user doesn't have to deal with 
>  > what's computer friendly - but reather what makes sense to the user. 
>  >  From a user's perspective if you have not rights to access a file then 
>  > why should you be allowed to delete it?
> 
> Because in Unix a name is not an attribute of a file.
 
 there is no excuse.

 selinux has already provided an alternative that is similar to NW
 file permissions.

 so there is no excuse.

 think about this.

 in what way is it possible for linux to fully support the NTFS
 filesystem?

 bearing in mind that you will need to communicate a little bit more
 than, but the direct equivalent of unix uid gid and secondary groups,
 from userspace into kernelspace - just like reading /etc/passwd but
 instead reading from a userspace daemon.

 l.

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:
 > On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:
 > 
 > > Marc Perkel writes:
 > > 
 > > [...]
 > > 
 > >  > Right - that's Unix "inside the box" thinking. The idea is to make the 
 > >  > operating system smarter so that the user doesn't have to deal with 
 > >  > what's computer friendly - but reather what makes sense to the user. 
 > >  >  From a user's perspective if you have not rights to access a file then 
 > >  > why should you be allowed to delete it?
 > > 
 > > Because in Unix a name is not an attribute of a file.
 >  
 >  there is no excuse.
 > 
 >  selinux has already provided an alternative that is similar to NW
 >  file permissions.

That's exactly the point: Unix file system model is more flexible than
alternatives. So that one can emulate foreign semantics on top of
it. But not other way around.

Nikita.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 02:30:44PM +0400, Nikita Danilov wrote:
> Luke Kenneth Casson Leighton writes:
>  > On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:
>  > 
>  > > Marc Perkel writes:
>  > > 
>  > > [...]
>  > > 
>  > >  > Right - that's Unix "inside the box" thinking. The idea is to make the 
>  > >  > operating system smarter so that the user doesn't have to deal with 
>  > >  > what's computer friendly - but reather what makes sense to the user. 
>  > >  >  From a user's perspective if you have not rights to access a file then 
>  > >  > why should you be allowed to delete it?
>  > > 
>  > > Because in Unix a name is not an attribute of a file.
>  >  
>  >  there is no excuse.
>  > 
>  >  selinux has already provided an alternative that is similar to NW
>  >  file permissions.
> 
> That's exactly the point: Unix file system model is more flexible than
> alternatives. 

 *grin*.  sorry - i have to disagree with you (but see below).

 i was called in to help a friend of mine at EDS to do a bastion sftp
 server to write some selinux policy files because POSIX filepermissions
 could not fulfil the requirements.

 the requirements were two-fold:

 1) for an authorised uploader to be able to copy files into
 a subdirectory but to be banned from seeing other files and
 banned from overwriting or deleting anything - existing or
 uploaded.  once the file's there, it's there.

 2) for an authorised downloader to be able to copy files _out_
 of a subdirectory and to be able to delete those files but
 to be banned from overwriting files, or modifying files,
 or adding files to the subdirectory.

 it is not possible to fulfil those requirements with POSIX - not even
 POSIX ACLs not even messing about creating groups with different rwx
 permissions.

 however, selinux was, because you have separate permissions for
 directories to create and remove names, read and write to the
 directory, and separate permissions _again_ for files.

 basically, i think (without checking) that there is one
 selinux permission per each inode operation and per each
 dnode operation totalling somewhere around thirty separate
 selinux filesystem permissions, which is _much_ better than
 the pathetic POSIX rwx and implicit directory rules stuff.

 POSIX permissions were designed to fit into what... 16 bits,
 so they didn't have a lot to play with.

 if i was to agree with you, it would be that the linux filesystem
 API is more flexible than the alternatives, most definitely not the
 Unix filesystem model (if you believe that to be POSIX).

 but the linux filesystem API i do not believe to be _as_ flexible as
 say the NTFS filesystem API (or the Netware one, although i don't know
 much about that one).

 l.

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:

[...]

 > > That's exactly the point: Unix file system model is more flexible than
 > > alternatives. 
 > 
 >  *grin*.  sorry - i have to disagree with you (but see below).
 > 
 >  i was called in to help a friend of mine at EDS to do a bastion sftp
 >  server to write some selinux policy files because POSIX filepermissions
 >  could not fulfil the requirements.

First, I was talking about flexibility attained through the separation
of notions of file and index. You just claimed elsewhere that this is
the direction ntfs took (with the introduction of hard-links).

Then, every security model has its weakness and corner cases. Try to
express

        rw-r-xrw- (0656)

POSIX bits with canonical NT ACLs (hint: in NT allow-ACEs are
accumulated).

[...]

 > 
 >  POSIX permissions were designed to fit into what... 16 bits,
 >  so they didn't have a lot to play with.

That very good property for a security model: simplicity is a virtue
here.

Nikita.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 04:17:35PM +0400, Nikita Danilov wrote:
> Luke Kenneth Casson Leighton writes:
> 
> [...]
> 
>  > > That's exactly the point: Unix file system model is more flexible than
>  > > alternatives. 
>  > 
>  >  *grin*.  sorry - i have to disagree with you (but see below).
>  > 
>  >  i was called in to help a friend of mine at EDS to do a bastion sftp
>  >  server to write some selinux policy files because POSIX filepermissions
>  >  could not fulfil the requirements.
> 
> First, I was talking about flexibility attained through the separation
> of notions of file and index. 

 oh, right.

> You just claimed elsewhere that this is
> the direction ntfs took 

 with a leap of a few steps, possibly: certainly directly i don't
 remember doing so.

> (with the introduction of hard-links).

 
> Then, every security model has its weakness and corner cases. Try to
> express
> 
>         rw-r-xrw- (0656)
> 
> POSIX bits with canonical NT ACLs (hint: in NT allow-ACEs are
> accumulated).
 
 they used not to be.  accumulative inherited ACLs were introduced
 in NT 5.0.

 and is accumulated ACLs such a bad thing?  it's certainly more
 space-efficient and administrative-efficient.

 l.

Re: what's next for the linux kernel?

[Horst von Brand]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:
> > Marc Perkel writes:

> > [...]
> > 
> >  > Right - that's Unix "inside the box" thinking. The idea is to make the 
> >  > operating system smarter so that the user doesn't have to deal with 
> >  > what's computer friendly - but reather what makes sense to the user. 
> >  >  From a user's perspective if you have not rights to access a file then 
> >  > why should you be allowed to delete it?

> > Because in Unix a name is not an attribute of a file.

>  there is no excuse.

It's not an excuse, it's part of a coherent view of how things work. Just
as Netware used to have its, and DOS had its (sort of). As the world view
underneath Unix, it is darn hard to "fix".

[This discussion sounds quite a lot like it is /you/ who needs "fixing",
 i.e., first wrap your head around Unix' ways...]

>  selinux has already provided an alternative that is similar to NW
>  file permissions.

Nope. SELinux provides MAC, i.e., mechanisms by which system-wide policy
(not the random owner of an object) ultimately decides what operations to
allow. That is not "file permissions". And yes, this is quite un-Unix-like.

[...]

>  in what way is it possible for linux to fully support the NTFS
>  filesystem?

If you ask me, preferably not at all, just let that unholy mess quietly go
the way of the dinosaurs. Sadly, interoperability is required at times,
so...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 02:47:27PM -0400, Horst von Brand wrote:
> Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> > On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:
> > > Marc Perkel writes:
> 
> > > [...]
> > > 
> > >  > Right - that's Unix "inside the box" thinking. The idea is to make the 
> > >  > operating system smarter so that the user doesn't have to deal with 
> > >  > what's computer friendly - but reather what makes sense to the user. 
> > >  >  From a user's perspective if you have not rights to access a file then 
> > >  > why should you be allowed to delete it?
> 
> > > Because in Unix a name is not an attribute of a file.
> 
> >  there is no excuse.
> 
> It's not an excuse, it's part of a coherent view of how things work. Just
> as Netware used to have its, and DOS had its (sort of). As the world view
> underneath Unix, it is darn hard to "fix".
> 
> [This discussion sounds quite a lot like it is /you/ who needs "fixing",
>  i.e., first wrap your head around Unix' ways...]
 
 asking "ordinary" people to do that is unrealistic: surely you know
 that?
 
 i just spent two hours helping a friend who wasn't familiar
 with the concept of "give root password for maintenance or
 press ctrl-d" they'd been pressing ctrl-d because it said so
 and now i'm going to have a 5-hour round-trip journey and possibly
 an overnight stay to sort out the mess.

> >  selinux has already provided an alternative that is similar to NW
> >  file permissions.
> 
> Nope. SELinux provides MAC, 

 yes.

> i.e., mechanisms by which system-wide policy
> (not the random owner of an object) ultimately decides what operations to
> allow. 

 yes.  the concept is not incompatible with what i said: the only bit
 that is wrong with what you've said is the word "Nope".

> That is not "file permissions". 

 part of the coverage of the MAC is file and directory permissions, and
 as i mentioned earlier, it so happens that each selinux permission
 corresponds, i believe one-to-one, with a function in the dnode and
 inode vfs higher-order-function tables in the linux kernel.

 example permissions (from postfix.te, policy source version 18):

	allow postfix_$1_t { sbin_t bin_t }:dir r_dir_perms;
	allow postfix_$1_t { bin_t usr_t }:lnk_file { getattr read };
	allow postfix_$1_t shell_exec_t:file rx_file_perms;

 i am confident enough with selinux to say that those are file
 and directory permissions.

 (r_dir_perms is a macro that expands to directory read
 permissions { read getattr lock search ioctl }, and
 rx_file_perms is a macro that expands to { read getattr lock
 execute ioctl })

 what this is saying is that postfix_$whatever_t context is
 allowed to read the contents of /sbin and /bin; it's also
 allowed to know if symlinks in /bin and /usr actually exist,
 and also allowed to follow those symlinks; and it's also allowed to
 know if shell-scripts exist, and also to read and ultimately
 execute them.

> And yes, this is quite un-Unix-like.

 this is a good thing.

> [...]
> 
> >  in what way is it possible for linux to fully support the NTFS
> >  filesystem?
> 
> If you ask me, preferably not at all, just let that unholy mess quietly go
> the way of the dinosaurs. Sadly, interoperability is required at times,
> so...

 *sigh*, tell me about it.  well, when reactos gets its NTFS driver, i
 will be sure to let you know.  i promise :)

 l.

Re: what's next for the linux kernel?

[jmerkey]

This is the most entertaining thread I've seen for a longest time on 
this list. Someone needs to write down the list of suggestions.

Re: what's next for the linux kernel?

[Neil Brown]

On Wednesday October 5, jmerkey@utah-nac.org wrote:
> This is the most entertaining thread I've seen for a longest time on 
> this list. Someone needs to write down the list of suggestions.
> 

But isn't that exactly what is wrong with this whole thread?
People saying "someone needs to" instead of "I will" or better yet, "I
have"??

NeilBrown

Re: what's next for the linux kernel?

[jmerkey]

Neil Brown wrote:

>On Wednesday October 5, jmerkey@utah-nac.org wrote:
>  
>
>>This is the most entertaining thread I've seen for a longest time on 
>>this list. Someone needs to write down the list of suggestions.
>>
>>    
>>
>
>But isn't that exactly what is wrong with this whole thread?
>People saying "someone needs to" instead of "I will" or better yet, "I
>have"??
>
>NeilBrown
>
>  
>
Yep. AM should probably do it, then post the finalists and ask for a vote.

Jeff

Re: what's next for the linux kernel?

[David Leimbach]

On 10/5/05, Neil Brown <neilb@suse.de> wrote:
> On Wednesday October 5, jmerkey@utah-nac.org wrote:
> > This is the most entertaining thread I've seen for a longest time on
> > this list. Someone needs to write down the list of suggestions.
> >
>
> But isn't that exactly what is wrong with this whole thread?
> People saying "someone needs to" instead of "I will" or better yet, "I
> have"??

Or perhaps they just need UTF-8-en_sarcasm so that we can better
detect it in his emails :-).

Dave

Re: what's next for the linux kernel?

[Horst von Brand]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Wed, Oct 05, 2005 at 02:47:27PM -0400, Horst von Brand wrote:
> > Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> > > On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:
> > > > Marc Perkel writes:

[...]

> > > > Because in Unix a name is not an attribute of a file.

> > >  there is no excuse.

> > It's not an excuse, it's part of a coherent view of how things work. Just
> > as Netware used to have its, and DOS had its (sort of). As the world view
> > underneath Unix, it is darn hard to "fix".
> > 
> > [This discussion sounds quite a lot like it is /you/ who needs "fixing",
> >  i.e., first wrap your head around Unix' ways...]

>  asking "ordinary" people to do that is unrealistic: surely you know
>  that?

And asking ordinary people to understand the (much more complex and opaque)
idea of "inheriting permissions from directories (sometimes, unless...)" is
surely much easier...

>  i just spent two hours helping a friend who wasn't familiar
>  with the concept of "give root password for maintenance or
>  press ctrl-d" they'd been pressing ctrl-d because it said so
>  and now i'm going to have a 5-hour round-trip journey and possibly
>  an overnight stay to sort out the mess.

Any suggestion for a better message?

[...]

>  example permissions (from postfix.te, policy source version 18):
> 
> 	allow postfix_$1_t { sbin_t bin_t }:dir r_dir_perms;
> 	allow postfix_$1_t { bin_t usr_t }:lnk_file { getattr read };
> 	allow postfix_$1_t shell_exec_t:file rx_file_perms;
> 
>  i am confident enough with selinux to say that those are file
>  and directory permissions.

OK, now I know for sure you are just an elaborate troll. SELinux is
/harder/ to grasp than Unix permissions, and /requires/ you to grasp them
as foundation.

[...]

> > >  in what way is it possible for linux to fully support the NTFS
> > >  filesystem?
> > 
> > If you ask me, preferably not at all, just let that unholy mess quietly go
> > the way of the dinosaurs. Sadly, interoperability is required at times,
> > so...
> 
>  *sigh*, tell me about it.  well, when reactos gets its NTFS driver, i
>  will be sure to let you know.  i promise :)

Great. Just keep in mind that time wasted on LKML is time taken away from
NTFS for ReactOS.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 11:06:05PM -0400, Horst von Brand wrote:
> >  i just spent two hours helping a friend who wasn't familiar
> >  with the concept of "give root password for maintenance or
> >  press ctrl-d" they'd been pressing ctrl-d because it said so
> >  and now i'm going to have a 5-hour round-trip journey and possibly
> >  an overnight stay to sort out the mess.
> 
> Any suggestion for a better message?
 
 warnings about "continuing to use your system without
 repairing the filesystem will result in further filesystem
 damage, ultimately making your system completely unusable.
 DO NOT ignore this message and press ctrl-d in order to
 continue using your system unless you REALLY know what you
 are doing."

 that sort of thing would have stopped my friend in their
 tracks and got them to phone me.

 l.

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

On Thu, 06 Oct 2005 00:03:09 BST, Luke Kenneth Casson Leighton said:
> On Wed, Oct 05, 2005 at 02:47:27PM -0400, Horst von Brand wrote:

> > Nope. SELinux provides MAC, 
> 
>  yes.
> 
> > i.e., mechanisms by which system-wide policy
> > (not the random owner of an object) ultimately decides what operations to
> > allow. 
> 
>  yes.  the concept is not incompatible with what i said: the only bit
>  that is wrong with what you've said is the word "Nope".
> 
> > That is not "file permissions". 
> 
>  part of the coverage of the MAC is file and directory permissions, and
>  as i mentioned earlier, it so happens that each selinux permission
>  corresponds, i believe one-to-one, with a function in the dnode and
>  inode vfs higher-order-function tables in the linux kernel.
> 
>  example permissions (from postfix.te, policy source version 18):
> 
> 	allow postfix_$1_t { sbin_t bin_t }:dir r_dir_perms;
> 	allow postfix_$1_t { bin_t usr_t }:lnk_file { getattr read };
> 	allow postfix_$1_t shell_exec_t:file rx_file_perms;
> 
>  i am confident enough with selinux to say that those are file
>  and directory permissions.

The part that you managed to miss is that this is MAC - *Mandatory*
Access Control.  This means that the *sysadmin* gets to say "this user
can't look at that file" - and there's nothing(*) either the owner of the
file or the user can do about it.  There's no chmod or chattr or chacl
command that the owner can issue to let somebody else read it - that's
the whole *point* of MAC.

(*) Well.. almost nothing.  The owner *may* be able to copy the contents
of the file to another file that the other user is allowed to read.  On the
other hand, the ability to do this would generally indicate a buggy policy....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Helge Hafting]

Valdis.Kletnieks@vt.edu wrote:

>The part that you managed to miss is that this is MAC - *Mandatory*
>Access Control.  This means that the *sysadmin* gets to say "this user
>can't look at that file" - and there's nothing(*) either the owner of the
>file or the user can do about it.  There's no chmod or chattr or chacl
>command that the owner can issue to let somebody else read it - that's
>the whole *point* of MAC.
>
>(*) Well.. almost nothing.  The owner *may* be able to copy the contents
>of the file to another file that the other user is allowed to read.  On the
>other hand, the ability to do this would generally indicate a buggy policy....
>  
>
Seems to me there is no use taking away the owners ability to chmod,
precisely because the owner always can get around that. (Unless
the owner doesn't even have the right to read his own file.)

You can have a restricted "copy" program, but nothing prevents a
user from making his own copy program, if he can read the file.
Or the user can load the file into the intended app, and "save as"
to some other filename and directory.  Or the user can do a screendump,
or even take a picture of the screen. 

Company policy may of course forbid the user to bring a camera, just as it
might forbid the user to do "chmod o+r" on important files.  I am not
sure that we need the OS to try to enforce such things. 

Helge Hafting

Re: what's next for the linux kernel?

[Horst von Brand]

Helge Hafting <helge.hafting@aitel.hist.no> wrote:
> Valdis.Kletnieks@vt.edu wrote:
> >The part that you managed to miss is that this is MAC - *Mandatory*
> >Access Control.  This means that the *sysadmin* gets to say "this user
> >can't look at that file" - and there's nothing(*) either the owner of the
> >file or the user can do about it.  There's no chmod or chattr or chacl
> >command that the owner can issue to let somebody else read it - that's
> >the whole *point* of MAC.
> >
> >(*) Well.. almost nothing.  The owner *may* be able to copy the contents
> >of the file to another file that the other user is allowed to read.  On the
> >other hand, the ability to do this would generally indicate a buggy policy....

> Seems to me there is no use taking away the owners ability to chmod,
> precisely because the owner always can get around that. (Unless
> the owner doesn't even have the right to read his own file.)

No. The point is that a (correct, complete) policy will prevent the user
from copying the contents to a file with less protection, by any means. No,
I did emphatically /not/ try to imply this is easy to set up (or use).

[...]

> Company policy may of course forbid the user to bring a camera, just as it
> might forbid the user to do "chmod o+r" on important files.  I am not
> sure that we need the OS to try to enforce such things.

If you don't trust your (typically fat-fingered) users and sysadmins...
Besides, the point behind the targeted policy in Red Hat/Fedora is to
forbid certain daemons to do nasty stuff. It is an additional protection
against misconfiguration or processes taken over by crackers.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]

On Thu, 06 Oct 2005 11:31:59 +0200, Helge Hafting said:

> You can have a restricted "copy" program, but nothing prevents a
> user from making his own copy program, if he can read the file.

Right, but a properly functioning and sufficiently fascist MAC system won't let
the user create a file in other security contexts that can be read by people
outside that context. See the recent work on supporting MLS (multi-level
security) and MCS (multi-category) in the SELinux tree...

> Or the user can load the file into the intended app, and "save as"

Again, "save as" doesn't create a file the other person can read unless the
other person is authorized.

> to some other filename and directory.  Or the user can do a screendump,

You *did* know that there's X hooks for Selinux, so that unauthorized applications
can't snarf the pixels of somebody else's window, right? ;)

> or even take a picture of the screen.

Not much we can do here.  On the other hand, it certainly creates a very low
upper limit on how much info you can extract how quickly... ;)

> Company policy may of course forbid the user to bring a camera, just as it
> might forbid the user to do "chmod o+r" on important files.  I am not
> sure that we need the OS to try to enforce such things. 

No, that is indeed outside the kernel's area.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 01:24:12PM +0400, Nikita Danilov wrote:

> One thing is clear: it's too late to fix Netware. Why should Unix
> emulate its lethal defects?
 
 the "lethal defects" of netware i believe could be more fairly
 attributed to it being proprietary software and the consequences
 thereof.

 that doesn't mean that you can't learn from it - the good bits,
 the bad bits and the ugly bits.

 l.

Re: what's next for the linux kernel?

[Marc Perkel]

Nikita Danilov wrote:

>Marc Perkel writes:
>
>[...]
>
> > Right - that's Unix "inside the box" thinking. The idea is to make the 
> > operating system smarter so that the user doesn't have to deal with 
> > what's computer friendly - but reather what makes sense to the user. 
> >  From a user's perspective if you have not rights to access a file then 
> > why should you be allowed to delete it?
>
>Because in Unix a name is not an attribute of a file.
>
>Files are objects that you read, write and truncate. They are
>represented by inodes.
>
>Separately from that, there is an indexing structure: directory
>tree. Directories map symbolical names to inodes. Obviously, adding a
>reference to an index, or removing it from one requires access
>permission to the _index_ rather then to the object being referenced.
>
>That two-level model of files and indexing on top of them is essential
>to Unix due to the flexibility and conceptual economy it provides.
>  
>
Now of you think "outside" the Linux box" you can see where people in 
the real world would expect that if you have no rights to a file to read 
or write to it that you shouldn't be able to delete it. In the outside 
world it's "duh - of course"! but for thouse that are in the "Unix Cult" 
you can't think past inodes.

> > 
> > Now - the idea is to create choice. If you need to emulate Unix nehavior 
> > for compatibility that's fine. But I would migrate away from that into a 
> > permissions paradygme that worked like Netware.
>
>And there are people believing that ITS (or VMS, or <insert your first
>passion here>...) set the standard to follow. :-)
>
>[...]
>
> > 
> > So - the thread is about the future so I say - time to fix Unix.
>
>One thing is clear: it's too late to fix Netware. Why should Unix
>emulate its lethal defects?
>
>Nikita.
>  
>

Once you'be had Netware permissions - even 1990 Netware permission - you 
are spoiled and everything else isn't even close.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Nikita Danilov]

Marc Perkel writes:
 > 

[...]

 > Now of you think "outside" the Linux box" you can see where people in 
 > the real world would expect that if you have no rights to a file to read 
 > or write to it that you shouldn't be able to delete it. In the outside 
 > world it's "duh - of course"! but for thouse that are in the "Unix Cult" 
 > you can't think past inodes.

Deleting files without read/write access to them is exactly what happens
in the real world of classified information: people who physically burn
paper folders have no right to open them. :-)

Please understand one simple thing: unlink(2) system call does not
_remove_ file. It just removes one of possibly many references to this
file from an index. To erase (a part of) a file body, one uses
truncate(2) system call that --wonders!-- requires write access to the
file.

[...]

 > 
 > Once you'be had Netware permissions - even 1990 Netware permission - you 
 > are spoiled and everything else isn't even close.
 > 

Repeating "sugar" doesn't make it sweet.

Nikita.

Re: what's next for the linux kernel?

[Helge Hafting]

On Wed, Oct 05, 2005 at 06:21:43AM -0700, Marc Perkel wrote:
> 
> Now of you think "outside" the Linux box" you can see where people in 
> the real world would expect that if you have no rights to a file to read 
> or write to it that you shouldn't be able to delete it. In the outside 
> world it's "duh - of course"! but for thouse that are in the "Unix Cult" 
> you can't think past inodes.
> 
On the contrary.  There are read & write permissions, and there is a
separate delete permission.  What is so hard to understand about that?
(In unix, "delete permission" is implemented as a write permission on
the directory that file resides in.)  

Please get rid of your "real world" notion.  There is no need to
assume that delete permission is tied to write permission, just
because _some_ os'es are like that.  Some are not that way too,
just get over that.  

And there is nothing strange about it, if we move outside the
"computer" box.  I have various paper documents with other people's
signatures on them.  I am free to shred them if I like (although
that might be bad for business.)  But I have no kind of write permission,
altering a signed document is fakery - a criminal activity.
So - permission to delete read-only documents predates electronic
file systems, and is a concept most "real-world" people understand quite well.

Helge Hafting

Re: what's next for the linux kernel?

[Tim Bird]

Nikita Danilov wrote:
> Marc Perkel writes:
> 
> [...]
> 
>  > Right - that's Unix "inside the box" thinking. The idea is to make the 
>  > operating system smarter so that the user doesn't have to deal with 
>  > what's computer friendly - but reather what makes sense to the user. 
>  >  From a user's perspective if you have not rights to access a file then 
>  > why should you be allowed to delete it?
> 
> Because in Unix a name is not an attribute of a file.
> 
> Files are objects that you read, write and truncate. They are
> represented by inodes.
> 
> Separately from that, there is an indexing structure: directory
> tree. Directories map symbolical names to inodes. Obviously, adding a
> reference to an index, or removing it from one requires access
> permission to the _index_ rather then to the object being referenced.
> 
> That two-level model of files and indexing on top of them is essential
> to Unix due to the flexibility and conceptual economy it provides.
>

We should print that on post-it notes for grandmothers
to read when they are interacting with Unix file systems.

>  > 
>  > So - the thread is about the future so I say - time to fix Unix.
> 
> One thing is clear: it's too late to fix Netware. Why should Unix
> emulate its lethal defects?

Like NetWare's defect of it being intuitive and easy to
administer file system rights?  Hey, here's a thought.  Maybe
the operating system could actually exist to SERVE the human
instead of vice versa.
  -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: what's next for the linux kernel?

[D. Hazelton]

[-- Attachment #1.1: Type: text/plain, Size: 3155 bytes --]

On Wednesday 05 October 2005 05:49, Marc Perkel wrote:
> D. Hazelton wrote:
> >>Novell Netware type permissions. ACLs are a step in the right
> >>direction but Linux isn't any where near where Novell was back in
> >>1990. Linux lets you - for example - to delete files that you
> >> have no read or write access rights to.
> >
> >As someone else pointed out, this is because unlinking is related
> > to your access permissions on the parent directory and not the
> > file.
>
> Right - that's Unix "inside the box" thinking. The idea is to make
> the operating system smarter so that the user doesn't have to deal
> with what's computer friendly - but reather what makes sense to the
> user. From a user's perspective if you have not rights to access a
> file then why should you be allowed to delete it?

You're confusing concepts. In Unix unlinking a file is not the same as 
deleting it. As has already been said, to remove content from a file, 
you truncate it, which, no surprise, requires that you have write 
access to a file. Even in DOS deleting a file, unless you use a 
secure delete program, doesn't delete the file - it merely changes 
the name slightly and marks the chain of FAT cluster entries as 
usable. 

I've had the displeasure of having to fix a netware system that had 
been so fsked up by an admin that had been fired that it was easier 
for me to remove the volume and restore it from a backup. The problem 
was that he made a large number of files with the administrative 
account removed from the ACL's... And the same problem plagues 
(plagued? I haven't checked up on this in a while) NTFS. It is all to 
possible to create a bunch of files with "Administrator" and all 
other "Administrator" class users form the ACL's and then kill that 
user.

> Now - the idea is to create choice. If you need to emulate Unix
> nehavior for compatibility that's fine. But I would migrate away
> from that into a permissions paradygme that worked like Netware.

So provide a filesystem and a set of tools for that filesystem. Nobody 
is standing in your way and the Linux filesystem and block device 
layers are open enough that this is an easy (though not simple) task.

> I started with Netware and I'm spoiled. They had it right 15 years
> ago and Linux isn't any where near what I was with Netware and DOS
> in 1990. Once you've had this kind of permission power Linux is a
> real big step down.

Oh, so that explains it. You got used to one paradigm and haven't been 
able to adjust to another. Well, as I have previously said, go ahead 
and provide us with the work.

> So - the thread is about the future so I say - time to fix Unix.

Time to fix Unix? I doubt something seriously borked would have 
outlasted every other OS on the market. Unix was around before 
Netware and, IMHO, will be around a long time after the last 
adherents of NetWare are gone. (and with MS doing it's level best to 
kill NetWare with it's own shared filesystems and built-in networking 
this cannot be that far off. After all, Netware was developed to fill 
a vacancy in the MS world.)

DRH

[-- Attachment #1.2: 0xA6992F96300F159086FF28208F8280BB8B00C32A.asc --]
[-- Type: application/pgp-keys, Size: 1365 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 01:22:33AM +0000, D. Hazelton wrote:
> On Tuesday 04 October 2005 19:47, Marc Perkel wrote:

> As someone else pointed out, this is because unlinking is related to 
> your access permissions on the parent directory and not the file.
 
 that's POSIX.

 i trust that POSIX has not been hard-coded into the entire design of
 the linux kernel filesystem architecture _just_ because it's ... POSIX.

 l.

Re: what's next for the linux kernel?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Wed, 05 Oct 2005 11:09:42 BST, Luke Kenneth Casson Leighton said:
> On Wed, Oct 05, 2005 at 01:22:33AM +0000, D. Hazelton wrote:
> > On Tuesday 04 October 2005 19:47, Marc Perkel wrote:
> 
> > As someone else pointed out, this is because unlinking is related to 
> > your access permissions on the parent directory and not the file.
>  
>  that's POSIX.
> 
>  i trust that POSIX has not been hard-coded into the entire design of
>  the linux kernel filesystem architecture _just_ because it's ... POSIX.

No, what got hard-coded were the concepts of inodes as the actual description
of filesystem objects, directories as lists of name-inode pairs, and the whole
user/group/other permission thing.  "unlink depends on the directory
permissions not the object unlinked" has been the semantic that people depended
on ever since some code at Bell Labs started supporting tree-structured
directories and multiple hardlinks.  POSIX merely codified existing behavior in
this case.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 06:23:09AM -0400, Valdis.Kletnieks@vt.edu wrote:

> >  i trust that POSIX has not been hard-coded into the entire design of
> >  the linux kernel filesystem architecture _just_ because it's ... POSIX.
> 
> No, what got hard-coded were the concepts of inodes as the actual description
> of filesystem objects, directories as lists of name-inode pairs, and the whole
> user/group/other permission thing.  "unlink depends on the directory
> permissions not the object unlinked" has been the semantic that people depended
> on 

 fortunately, selinux has begun the path away from that kind of implicit
 ruling.

 l.

Re: what's next for the linux kernel?

[Nix]

On 4 Oct 2005, Marc Perkel announced authoritatively:
> Reiser 4 - The idea of building a file system on top of a database is
> the right way to go. Reiser is onto something here and this is a
> technology that needs to be built upon. It's current condition is a
> little on the week side - no ACLs for example - but the underlying
> concept is ound.

Well, it's not a `technology' (whatever that means). It's an idea, and
an implementation: a unified namespace built atop a database. Parts of
it make sense (the unified namespace: look at /sys for a bunch of fairly
recent unification); much of it makes sense but is hard to implement
(e.g. plugins); much makes conceptual sense but breaks POSIX in horrible
ways (files-as-directories, possibly meta-directories with arbitrary
content, at least if you're allowed to do all POSIX-allowed things to
that content: what happens to stat() now if some bugger removes a file's
creation time or size, or moves it out to some parent directory? Does
that even make sense? Whether or not it's allowed, you've broken a good
few existing programs.)

> Novell Netware type permissions.

... are an administrative *nightmare*. One of the major advantages of
the perhaps-overly-simplistic Unix permissions model is that a simple
`ls -l' can show you the complete permissions of a lot of files in a few
screensful, in an easily-comprehensible manner.

Ever tried doing that with a directory with most files having ACLs?
Until we have better userspace tools, across-the-board ACLs are a
non-starter. Now admittedly the iprovements are minor (e.g. highlighting
things with ACLs differing from the norm), but the fact remains that
they are *not here yet*.

>                                  ACLs are a step in the right
> direction but Linux isn't any where near where Novell was back in
> 1990. Linux lets you - for example - to delete files that you have no
> read or write access rights to. Netware on the other hand prevents you
> from deleting files that you can't write to and if you have no right
> it is as if the file isn't there. You can't even see it in the
> directory.

Oh, joy. So now I can't remove a writable directory if someone else has
created a file I can't write to in it, even if that directory is owned
by me. That sounds really unpleasant to me.

What next? Forbidding unlink() of running executables?

>             Netware also has inherited permissions like Windows and
> Samba has and this is doing it right.

s/right/wrong/

Look at a NetWare permission on some file and you can't tell what the
effective permission is, because it depends on inherited ones as well.
Look at an effective permission and you can't tell which parts of it
will change if you change the inherited permission. Change an inherited
permission and some of the inheritees' permissions might suddenly make
no sense. The UI errs on the side of showing you effective permissions,
which makes most sense, but is still far from ideal.

Unix has inherited permissions in one sense: you can't get at a file
via some path if one of the directories in that path is unreadable.
Even *that* causes regular problems, such that sendmail (for instance)
needs special code to check for this case.

And you want to make this *more* complex?

Remember: complexity is the enemy of security.

>                                       File systems and individual
> directories should be able to be flagged as casesensitive/insensitive.

Only if you're willing to change POSIX to include a call to check filenames
for identity, and rewrite every POSIX app to use such a call where filenames
are compared, *and* discard one or more of the following (I think you can't
avoid losing at least two, actually):

- user-settable locales via LC_* and LANG
- the property that directories may contain only one file of a given name
- the property that a rename from name A to name B and then back to name A
  again yields a file with the same name it started with
- the property that open ("foo", ... | O_CREAT) yields a file named `foo'
  for all `foo'

I doubt that losing any of these properties would be considered
acceptable, and as for rewriting every app on earth that does filename
comparison, no chance.

It would also require case-conversion and locale-handling code, probably
including UTF-8 canoncalization code, inside the kernel. This would
greatly increase kernel complexity for a very small reward, and lead to
Al Viro's early death from cerebral aneurysm combined with involuntary
projectile vomiting. This cannot be considered a good thing.

>                            Permissions need to be fine grained and
> easy to use.

This is an impossible combination, especially if you add `secure' to
that list. Fine-grained capabilities on executable files, now *that*
is a nice thing to mostly-replace the odious setuid bit with.

> The bootup sequence of Linux is pathetic. What an ungodly mess. The

`The' bootup sequence? There are a number of alternative init systems
under development, some in use by reasonably major distros.

> FSTAB file needs to go and a smarter system needs to be developed. I

What's wrong with it? Sure, it doesn't solve all problems
(e.g. automounting; that's why we have automounters), and some of its
fields are passing obsolete (e.g.  the fs_freq field), but that doesn't
make it *bad*. Those problems it aims to solve, it solves well.

Now /etc/mtab, *that* is an abomination, and a small kernel improvement
(allowing arbitrary flag strings to be passed by mount into the kernel
solely for display in the appropriate /proc/mounts field) could
eliminate it and replace it entirely with /proc/mounts.

> I think development needs to be done to make the kernel cleaner and

This has long been a goal.

> smarter

`Smarter' is easy to *say*. Defining what it *means* is another matter.
Myself I consider the new pluggable I/O schedulers and pluggable
packet schedulers make the kernel `smarter'. Perhaps you do not.

>         rather than just bigger

This has never been a *goal*. It's a side-effect, that's all.

>                                 and faster.

Faster is important, especially on very small and very large hardware,
where unusual things can become bottlenecks.

>                                             It's time to look at what
> users need

What? Not in the kernel, it isn't. Users never see the
kernel. Sysadmins, sure; glibc, sure; users, no. Not even userspace
programs, except inasmuch as their requirements are reflected by new
demands on the kernel.

>            and try to make Linux somewhat more windows like in being
> able to smartly recover from problems.

In my experience Windows tries to smartly recover, fails, and implodes
in a heap, telling all and sundry to `contact your system
administrator'. When you *are* the system administrator, this is less
than helpful.

One thing that *would* be nice is an improvement on errno, which is
a terribly coarse-grained error handling system. Something like the
sa_siginfo field, so you can pass additional info back with errors...
but for now that can be kludged around by passing back errors by
filling up buffers passed in other parameters.

errno seems to be Good Enough, but it's... *primitive*.

(What did Plan 9 do? Error strings?)

>                                        Perhaps better error messages
> that your traditional kernel panic or hex dump screen of death.

The `traditional kernel panic' includes a backtrace; if you have kernel
syslogging or the serial console turned on, this can even work if
process scheduling and disk I/O are both dead.

Linux doesn't do `hex dump screens of death'.

A number of userspace tools like smartd provide nice warnings of some
critical non-kernel system failures in whatever way you see fit (my
systems send me an email and a page and shut themselves down if they
overheat or suffer a disk failure; much more elaborate things are
possible). But, again, this is not a kernel thing.

> The big challenge for Linux is to be able to put it in the hands of
> people who don't want to dedicate their entire life to understanding
> all the little quirks that we have become used to. The slogan should
> be "this just works"

This is the distributors' job (those who want to do that). One
disadvantage of `just works' systems is that they tend to be fiercely
complex and thus, when they *do* fail, the failures are correspondingly
hard to diagnose. (This is not always true: udev just works and is much
*less* complex than the system it replaces, with almost entirely
non-opaque failure modes. This comes, I think, of designing it properly
from the start. Kudos to Greg K-H!)

There will always be openings for less elaborate systems, I think.

>                      and is intuitive.

`The only intuitive interface is the nipple.'

> Linux Visionary

*chuckle*

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll

Re: what's next for the linux kernel?

[Rik van Riel]

On Tue, 4 Oct 2005, Marc Perkel wrote:

> Marc Perkel
> Linux Visionary

The problem I have with most visionaries is that while 
they see lots of stuff, they never show anything to the
rest of the world.

Unless you can explain to other people why your idea
makes sense, or unless you implement your idea, it won't
happen.

If the idea is good enough, either of explanation or
implementation should be enough.

Preaching, OTOH, does not convince people.

-- 
All Rights Reversed

Re: what's next for the linux kernel?

[Marc Perkel]

Rik van Riel wrote:

>On Tue, 4 Oct 2005, Marc Perkel wrote:
>
>  
>
>>Marc Perkel
>>Linux Visionary
>>    
>>
>
>The problem I have with most visionaries is that while 
>they see lots of stuff, they never show anything to the
>rest of the world.
>
>Unless you can explain to other people why your idea
>makes sense, or unless you implement your idea, it won't
>happen.
>
>If the idea is good enough, either of explanation or
>implementation should be enough.
>
>Preaching, OTOH, does not convince people.
>
>  
>
This stuff I'm talking about is not theoretical. It's been in Novell 
Netware since 1990 and it works great. Netware with DOS in 1990 is still 
far superior to Linux today. Once you've had Netware - Linux is 
laughable. All youhave to do is look ate Netware and copy it. Or the 
mars-nwe netware emulator for that matter. The code to do this already 
exists.

-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Al Viro]

On Wed, Oct 05, 2005 at 08:58:13AM -0700, Marc Perkel wrote:
> This stuff I'm talking about is not theoretical. It's been in Novell 
> Netware since 1990 and it works great. Netware with DOS in 1990 is still 
> far superior to Linux today. Once you've had Netware - Linux is 
> laughable. All youhave to do is look ate Netware and copy it. Or the 
> mars-nwe netware emulator for that matter. The code to do this already 
> exists.

Novell will happily sell you Netware if you are so inclined, I suppose.
As long as its consentual, it's really your business - one of three is
not that bad, even if "sane" and "safe" are missing...

Re: what's next for the linux kernel?

[Marc Perkel]

Al Viro wrote:

>On Wed, Oct 05, 2005 at 08:58:13AM -0700, Marc Perkel wrote:
>  
>
>>This stuff I'm talking about is not theoretical. It's been in Novell 
>>Netware since 1990 and it works great. Netware with DOS in 1990 is still 
>>far superior to Linux today. Once you've had Netware - Linux is 
>>laughable. All youhave to do is look ate Netware and copy it. Or the 
>>mars-nwe netware emulator for that matter. The code to do this already 
>>exists.
>>    
>>
>
>Novell will happily sell you Netware if you are so inclined, I suppose.
>As long as its consentual, it's really your business - one of three is
>not that bad, even if "sane" and "safe" are missing...
>  
>

That's not the point. The point is that Netware has a far superior 
permission system and I am suggesting the the Linux community learn from 
it and take advantage of seeing what better looks like and improving itself.


-- 
Marc Perkel - marc@perkel.com

Spam Filter: http://www.junkemailfilter.com
    My Blog: http://marc.perkel.com

Re: what's next for the linux kernel?

[Lennart Sorensen]

On Wed, Oct 05, 2005 at 09:23:56AM -0700, Marc Perkel wrote:
> That's not the point. The point is that Netware has a far superior 
> permission system and I am suggesting the the Linux community learn from 
> it and take advantage of seeing what better looks like and improving itself.

Linux is compatible with unix applications.  Netware is not.  Supporting
some useless netware feature at the expense of posix/unix compatibility
would be insane.

If you can't do it with unix permissions or unix permissions + ACL, you
don't need to do it at all most likely, and even more likely you
probably don't actually understand what you are trying to accomplish and
will probably end up making something insecure or broken instead.

Having dealt with netware, trying to administrate that mess was way to
painful and confusing.  What a horrible interface.  I can't imagine
anything I would want to borrow from netware.

Len Sorensen

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

On Wed, Oct 05, 2005 at 03:30:24PM -0400, Lennart Sorensen wrote:
> On Wed, Oct 05, 2005 at 09:23:56AM -0700, Marc Perkel wrote:
> > That's not the point. The point is that Netware has a far superior 
> > permission system and I am suggesting the the Linux community learn from 
> > it and take advantage of seeing what better looks like and improving itself.
> 
> Linux is compatible with unix applications.  Netware is not.  Supporting
> some useless netware feature at the expense of posix/unix compatibility
> would be insane.
> 
> If you can't do it with unix permissions or unix permissions + ACL, you
> don't need to do it at all most likely, and even more likely you

 the bastion sftp example i gave which required selinux on top of a much
 broader set of POSIX file permissions demonstrates the fallacy of your
 statement.

 try to achieve the same effect with POSIX - even POSIX ACLs
 (uploader only has create and write, not read, not delete;
  downloader has read and delete, not write, not create)

 and you will fail, miserably, because under POSIX, write implies
 create.

 l.

Re: what's next for the linux kernel?

[Nikita Danilov]

Luke Kenneth Casson Leighton writes:

[...]

 >  the bastion sftp example i gave which required selinux on top of a much
 >  broader set of POSIX file permissions demonstrates the fallacy of your
 >  statement.

Still waiting you to express 0656 with well-formed NT ACLs.

Nikita.

Re: what's next for the linux kernel?

[Joe Bob Spamtest]

Luke Kenneth Casson Leighton wrote:
>  the bastion sftp example i gave which required selinux on top of a much
>  broader set of POSIX file permissions demonstrates the fallacy of your
>  statement.
> 
>  try to achieve the same effect with POSIX - even POSIX ACLs
>  (uploader only has create and write, not read, not delete;
>   downloader has read and delete, not write, not create)
> 
>  and you will fail, miserably, because under POSIX, write implies
>  create.

you, however, seem to be missing the point that these are special 
circumstances. in 99% of all cases, regular unix file permissions are 
sufficient. when you start needing special silly permissions for things 
like this, we have special silly tools to accommodate you. Use them. 
Deal with it.

adding a permissions schema similar to that found in windows/netware 
would only unneccessarily complicate things, and most likely end up 
breaking everything.

bottom line: if you want to see support like this in linux, write a 
filesystem with these capabilities built-in. If you don't want to/can't 
write it, then stop complaining and continue to use netware (read: shit 
or get off the pot).

Re: what's next for the linux kernel?

[Joe Bob Spamtest]

Marc Perkel wrote:
> This stuff I'm talking about is not theoretical. It's been in Novell 
> Netware since 1990 and it works great. Netware with DOS in 1990 is still 
> far superior to Linux today. Once you've had Netware - Linux is 
> laughable. All youhave to do is look ate Netware and copy it. Or the 
> mars-nwe netware emulator for that matter. The code to do this already 
> exists.

If Linux is "laughable", and netware is "far superior", then get your 
ass back on a netware box and stop complaining. If you like it so much, 
use it. If you don't like the way linux works, don't use it. End of story.

Re: what's next for the linux kernel?

[Luke Kenneth Casson Leighton]

please can we move this discussion _to_ the linux visionaries
mailing list?

i for one would actually like a place where linux kernel developers can
say, instead of "get lost, xxxxhead, you're wasting our time" or
patronise people with IQs > 140 with "go read the kernel newbies faq":

	"Hmm, sounds like you have a baad case of linux vision, son.
	Go Preach it on LVML, and may you ever love the Penguin."

anyone willing to host such a list at a _real_ server please
speak up now or forever hold your peace.

l.

On Wed, Oct 05, 2005 at 11:54:03AM -0400, Rik van Riel wrote:

> On Tue, 4 Oct 2005, Marc Perkel wrote:
> 
> > Marc Perkel
> > Linux Visionary
> 
> The problem I have with most visionaries is that while 
> they see lots of stuff, they never show anything to the
> rest of the world.
> 
> Unless you can explain to other people why your idea
> makes sense, or unless you implement your idea, it won't
> happen.
> 
> If the idea is good enough, either of explanation or
> implementation should be enough.
> 
> Preaching, OTOH, does not convince people.
> 
> -- 
> All Rights Reversed

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--