software packaging and ReiserFS v4

software packaging and ReiserFS v4

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

I've got this thought that ReiserFS v4 may make it possible to
implement a _really_ trivial software package management system,
with even better power than rpm.

I'm building myself a new Linux distrib; my proximate motivation is
that while there's lots I like about Red Hat --- in particular, rpm
is the best software package mgmt tool I've used or heard of ---
there's still a lot of accumulated cruft there.

As I've gotten deeper into the project, I've had my nose rubbed in
the fact that rpm is the accumulating cruft heart; it's the worst.
Way, way overdue for a rewrite.

So I started fantasizing about the replacement. Some bits are easy;
rather than trying to write a parser for yet another poorly
documented subtle and complex file format like rpm spec files, the
spec file will simply be a Lua <URL:http://www.lua.org/> source file
that exports the necessary data.

Then I started thinking about the database that a good software
packging tool needs to maintain. After a bit, it hit me that
ReiserFS 4 would be _killer_!

Let the package install process deposit the "spec file" (lua data
decl) in /var/lib/pkg/{pkgname}. Addenda like e.g. the checksums,
and pre- and post-install scripts can be
/var/lib/pkg/{pkgname}/sha1, /var/lib/pkg/{pkgname}/pre,
/var/lib/pkg/{pkgname}/post, etc. Whatever open-ended addenda the
package should wish to keep track of on a per-package basis can be
simply added there.

The one tricky bit --- and it's the performance downfall of the
current rpm --- is the reverse-mapping database, the one that maps a
pathname of an installed file to the package that installed it.
Maintaining that, and ensuring its consistency with the forward
database, makes rpm slow.

With ReiserFS v4 its as natural as can be; a file /usr/bin/foo,
installed by a package foo, would simply have a symlink

	/usr/bin/foo/pkg -> /var/lib/pkg/foo

Besides scaling elegantly and staying wonderfully simple, this has
the additional otherwise-impossible feature that the package
attribution automatically correctly updates if you move the file
(although the forward mappings within /var/lib/pkg/foo wouldn't
update automagically).

I'm continuing to do the preliminary, basic dev of my new distrib on
Reiser 3 w/ rpm, because they're what I've got, but when v4 comes
around I'm seriously hoping to re-wrap the whole schmeer taking full
advantage of v4 to make a very simple new package manager.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: software packaging and ReiserFS v4

[Jürgen Botz]

Bennett Todd wrote:
> As I've gotten deeper into the project, I've had my nose rubbed in
> the fact that rpm is the accumulating cruft heart; it's the worst.

Amen.  It's so bad that RH can't even fix it anymore... It took
them the better part of a year to fix the widely reported hanging
bug that appeared in RH 8.0.

> Then I started thinking about the database that a good software
> packging tool needs to maintain. After a bit, it hit me that
> ReiserFS 4 would be _killer_!

Uh-huh.  I've been dreaming about something like this for 10 years,
even since I first played around with the Prospero file system.
In its implementation Prospero fell short of its promise, but it
demonstrated the revolutionary idea that closure enabled complete
customization of namespaces in a file system.

> With ReiserFS v4 its as natural as can be; a file /usr/bin/foo,
> installed by a package foo, would simply have a symlink
> 
> 	/usr/bin/foo/pkg -> /var/lib/pkg/foo
> 
> Besides scaling elegantly and staying wonderfully simple, this has
> the additional otherwise-impossible feature that the package
> attribution automatically correctly updates if you move the file
> (although the forward mappings within /var/lib/pkg/foo wouldn't
> update automagically).

They could, with the right plug-in.  Without such a plug-in I don't
see why you need reiser4 for this... in fact, this kind of scheme
has already been implemented in some older package managers.  There
are several LISA papers talking about this kind of thing since
the early 90s.  In particular the CMU "depot" system comes to mind.

But take this idea further.... in my ideal system there is no /usr.
/usr is just a historical artifact, anyway.

Nor is there a $HOME.  Depending on your point of view (more about
that below) there is only /, /bin, /lib, /etc, and maybe /man or
/doc, plus any directories you create.  Your files go into /.  Or
at least that's what you think...

In /bin are the programs you use.  There are all the standard Unix
utils, of course, but if you're an emacs user there's /bin/emacs, and
if you're a vi user there's /bin/vi.  That's for you as a user... for
root all commands that are installed in the system are in /bin.  If
you become a heretic ;-) and convert from emacs to vi, you just use
a UI that lets you deselect the emacs package from being included in
your namespace and select the vi package.

The key to this are what we might call "perspectives".  Every user
has their own perspective, or maybe more than one that they can
chose between.  There's also a special perspective that
shows every package in its own hierarchy, i.e.
/pkg/emacs/20.1/{bin,lib,man}.  This perspective is your "package
manager" if you will.  In addition every execution context
has its own perspective... when emacs runs it always sees itself and
all its files as "/pkg/emacs/20.1/...".  This is closure, and
it guarantees that every reference can always be resolved.  It once
and for all solves the "hardcoded path" problem.  It also resolves
name collision problems, multiple installed package version problems,
etc., etc.

Reiser4 as it is today can't do this yet.  But it provides the
infrastructure on which such a system (and others) can be built.

I have learned that very few people actually understand the importance
and power of flexible namespaces, never mind closure in namespaces.
Fortunately Hans Reiser does, and since his wonderful filesystem has
other benefits is being adopted in spite of people's ignorance of
its far-ranging implications!  ;-)

:j

--
Jürgen Botz               | While differing widely in the various
jurgen@botz.org           | little bits we know, in our infinite
                           | ignorance we are all equal. -Karl Popper

Re: software packaging and ReiserFS v4

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 2765 bytes --]

2003-09-03T17:18:07 J?rgen Botz:
> >With ReiserFS v4 its as natural as can be; a file /usr/bin/foo,
> >installed by a package foo, would simply have a symlink
> >
> >	/usr/bin/foo/pkg -> /var/lib/pkg/foo
> >
> >Besides scaling elegantly and staying wonderfully simple, this has
> >the additional otherwise-impossible feature that the package
> >attribution automatically correctly updates if you move the file
> >(although the forward mappings within /var/lib/pkg/foo wouldn't
> >update automagically).
> 
> They could, with the right plug-in.

A plug-in to reach in and edit the forward references in the package
database misses my vision of "simple"; I think I'd rather live
without it (after all, we do now) than live with it.

> Without such a plug-in I don't see why you need reiser4 for
> this... [...]

I'd have an executable, /usr/bin/wc, installed by the
textutils-2.0.21-5 package, and its reverse link would be

	/usr/bin/wc/pkg -> /var/lib/pkg/textutils-2.0.21-5

How would you do that without Reiser4's ability to pun the same
pathname as both a file and an associated directory?

> in fact, this kind of scheme has already been implemented
> in some older package managers.  There are several LISA papers
> talking about this kind of thing since the early 90s.  In
> particular the CMU "depot" system comes to mind.

A different vision; I prefer for my /usr/bin/wc to be a real file,
not a symlink to /usr/depot/textutils-2.0.21-5/bin/wc or whatever.
Different people, different tastes:-).

> But take this idea further.... in my ideal system there is no /usr.
> /usr is just a historical artifact, anyway.

It serves some pragmatic roles; distinguishing the minimum needed to
get booted up (/bin, /sbin, ...) from the stuff that's only needed
multiuser (/usr/bin, ...) is sometimes helpful. And a small,
rarely-accessed, readonly / is less prone to damage.

> Nor is there a $HOME.  Depending on your point of view (more
> about that below) there is only /, /bin, [...] The key to this
> are what we might call "perspectives".  Every user has their own
> perspective, or maybe more than one that they can chose between.
> [...]

I _think_ this was big in Plan 9, if I'm not mistaken.

Different folks are game for different things.

I'm rebellious enough to be fantasizing a nice simple software
package manager that can easily allow an rpm user to say "bpm"
instead and have all the same commands work --- or have all but a
couple of them trivially implementable using normal native system
commands for that matter --- by taking advantage of Reiserfs 4's
open-ended attributes dir stuff. But I'm not so sublime about
completely abandoning the Filesystem Heirarchy Standard.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: software packaging and ReiserFS v4

[Jürgen Botz]

Bennett Todd wrote:
>>>(although the forward mappings within /var/lib/pkg/foo wouldn't
>>>update automagically).
>>
>>They could, with the right plug-in.
> 
> A plug-in to reach in and edit the forward references in the package
> database misses my vision of "simple";

*sigh*

You do not seem to grasp the power of your own idea.  There is no
need for a separate "package database" with your scheme.  What is
the package database but the contents of your 'pkg' directory?
Your links (/usr/bin/program/pkg -> /pkg/program/...) provide the
inversion.  You only need a plug-in to make this "symbolic link"
something that gets atomically changed at both ends.  This is
nothing special... it is no more special than a regular symbolic
link, whose semantics are also provided by a plug-in in reiser4.

I can only paraphrase Hans: The filesystem *is* the database.

:j

--
Jürgen Botz               | While differing widely in the various
jurgen@botz.org           | little bits we know, in our infinite
                           | ignorance we are all equal. -Karl Popper

Re: software packaging and ReiserFS v4

[Grant Miner]

>>Nor is there a $HOME.  Depending on your point of view (more
>>about that below) there is only /, /bin, [...] The key to this
>>are what we might call "perspectives".  Every user has their own
>>perspective, or maybe more than one that they can chose between.
>>[...]
> 
> 
> I _think_ this was big in Plan 9, if I'm not mistaken.

Yes, this is what Plan 9 does.  Actually, Linux now has support for 
per-process namespaces (perspectives) too (it was added by Al Viro in 
2.5.2).  He also might may write code to do unions.

Re: software packaging and ReiserFS v4

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

2003-09-03T17:52:04 J?rgen Botz:
> Bennett Todd wrote:
> >>>(although the forward mappings within /var/lib/pkg/foo wouldn't
> >>>update automagically).
> >>
> >>They could, with the right plug-in.
> >
> >A plug-in to reach in and edit the forward references in the package
> >database misses my vision of "simple";
>
> There is no need for a separate "package database" with your
> scheme.  What is the package database but the contents of your
> 'pkg' directory?

The package database is the contents of /var/lib/pkg/ --- files,
each nearly completely documenting one package, with addenda as
needed under each --- plus the symlinks under each installed file
back representing the inverse map.

> Your links (/usr/bin/program/pkg -> /pkg/program/...) provide the
> inversion.  You only need a plug-in to make this "symbolic link"
> something that gets atomically changed at both ends.

The problem is that the forward maps aren't a symlink, they're text
files enumerating the contents of the package together with
attributes (is this a doc file, a config file, ...) and the
checksums table (a text file, w/ full pathnames and sha1 sums in
hex); that last usually signed by a detached OpenPGP sig.

> I can only paraphrase Hans: The filesystem *is* the database.

More than before, yes, but I don't quite see how it's practical for
it to be _every_ database.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: software packaging and ReiserFS v4

[Jeffrey Yasskin]

Bennett Todd wrote:

>2003-09-03T17:52:04 J?rgen Botz:
>  
>
>>Bennett Todd wrote:
>>    
>>
>>>>>(although the forward mappings within /var/lib/pkg/foo wouldn't
>>>>>update automagically).
>>>>>          
>>>>>
>>>>They could, with the right plug-in.
>>>>        
>>>>
>>>A plug-in to reach in and edit the forward references in the package
>>>database misses my vision of "simple";
>>>      
>>>
>>There is no need for a separate "package database" with your
>>scheme.  What is the package database but the contents of your
>>'pkg' directory?
>>    
>>
>
>The package database is the contents of /var/lib/pkg/ --- files,
>each nearly completely documenting one package, with addenda as
>needed under each --- plus the symlinks under each installed file
>back representing the inverse map.
>
>  
>
>>Your links (/usr/bin/program/pkg -> /pkg/program/...) provide the
>>inversion.  You only need a plug-in to make this "symbolic link"
>>something that gets atomically changed at both ends.
>>    
>>
>
>The problem is that the forward maps aren't a symlink, they're text
>files enumerating the contents of the package together with
>attributes (is this a doc file, a config file, ...) and the
>checksums table (a text file, w/ full pathnames and sha1 sums in
>hex); that last usually signed by a detached OpenPGP sig.
>  
>
I think you want the bidirectional links which are mentioned in the Link 
Taxonomy section of http://namesys.com/v4/v4.html (search for "bilink"). 
I don't think they're implemented yet, but once they are, the forward 
map will probably consist of a pseudo-file (..links?) in /var/lib/pkg/foo/








--Jeffrey

Re: software packaging and ReiserFS v4

[Richard Heycock]

On Thu, 2003-09-04 at 02:54, Bennett Todd wrote:
> I've got this thought that ReiserFS v4 may make it possible to
> implement a _really_ trivial software package management system,
> with even better power than rpm.
> 
> I'm building myself a new Linux distrib; my proximate motivation is
> that while there's lots I like about Red Hat --- in particular, rpm
> is the best software package mgmt tool I've used or heard of ---
> there's still a lot of accumulated cruft there.

Why not just use apt-get/dpkg? It make a mockery of rpms.

rgh

> 
> As I've gotten deeper into the project, I've had my nose rubbed in
> the fact that rpm is the accumulating cruft heart; it's the worst.
> Way, way overdue for a rewrite.
> 
> So I started fantasizing about the replacement. Some bits are easy;
> rather than trying to write a parser for yet another poorly
> documented subtle and complex file format like rpm spec files, the
> spec file will simply be a Lua <URL:http://www.lua.org/> source file
> that exports the necessary data.
> 
> Then I started thinking about the database that a good software
> packging tool needs to maintain. After a bit, it hit me that
> ReiserFS 4 would be _killer_!
> 
> Let the package install process deposit the "spec file" (lua data
> decl) in /var/lib/pkg/{pkgname}. Addenda like e.g. the checksums,
> and pre- and post-install scripts can be
> /var/lib/pkg/{pkgname}/sha1, /var/lib/pkg/{pkgname}/pre,
> /var/lib/pkg/{pkgname}/post, etc. Whatever open-ended addenda the
> package should wish to keep track of on a per-package basis can be
> simply added there.
> 
> The one tricky bit --- and it's the performance downfall of the
> current rpm --- is the reverse-mapping database, the one that maps a
> pathname of an installed file to the package that installed it.
> Maintaining that, and ensuring its consistency with the forward
> database, makes rpm slow.
> 
> With ReiserFS v4 its as natural as can be; a file /usr/bin/foo,
> installed by a package foo, would simply have a symlink
> 
> 	/usr/bin/foo/pkg -> /var/lib/pkg/foo
> 
> Besides scaling elegantly and staying wonderfully simple, this has
> the additional otherwise-impossible feature that the package
> attribution automatically correctly updates if you move the file
> (although the forward mappings within /var/lib/pkg/foo wouldn't
> update automagically).
> 
> I'm continuing to do the preliminary, basic dev of my new distrib on
> Reiser 3 w/ rpm, because they're what I've got, but when v4 comes
> around I'm seriously hoping to re-wrap the whole schmeer taking full
> advantage of v4 to make a very simple new package manager.
> 
> -Bennett
-- 
"It is possible to make things of great complexity out of things
 that are very simple. There is no conservation of simplicity"
 -- Stephen Wolfram

Richard Heycock <rgh@roughage.com.au>
tel : 0410 646 369
key fingerprint : 909D CBFA C669 AC2F A937 AFA4 661B 9D21 EAAB 4291

Re: software packaging and ReiserFS v4

[Grant Miner]

If you are making a packaging system, I recommend you do not allow 
scipts, since they open up a whole can of worms.  (They're impossible to 
invert, introduce security problems, make additional work for GUI 
package managers, and cause too many problems.)

Re: software packaging and ReiserFS v4

[Mike Fedyk]

On Fri, Sep 05, 2003 at 01:15:41PM -0500, Grant Miner wrote:
> If you are making a packaging system, I recommend you do not allow 
> scipts, since they open up a whole can of worms.  (They're impossible to 
> invert, introduce security problems, make additional work for GUI 
> package managers, and cause too many problems.)

What do you replace them with then?

Re: software packaging and ReiserFS v4

[Ragnar Kjørstad]

On Fri, Sep 05, 2003 at 12:44:13PM -0700, Mike Fedyk wrote:
> On Fri, Sep 05, 2003 at 01:15:41PM -0500, Grant Miner wrote:
> > If you are making a packaging system, I recommend you do not allow 
> > scipts, since they open up a whole can of worms.  (They're impossible to 
> > invert, introduce security problems, make additional work for GUI 
> > package managers, and cause too many problems.)
> 
> What do you replace them with then?

I think if scripts were to be removed from the packaging system you
would have to add a package-feature for every single thing that scripts
are currently used for. 

If you look through a collection of packages I think you will find that
they typically do things like:
- create users
- start / stop deamons
and so on.

So the package-system would need a way to specify that a user is part of
the package (that would be very easy with a /etc/passwd.d-system) and
that deamons that are part of the package should be started at install
and stopped at uninstall-time.

There are probably dozens of other examples of common operations - and
you would not be able to remove the script-feature unless 99% of what
they are used for is not needed anymore - so it's a complicated task.


-- 
Ragnar Kjørstad

Re: software packaging and ReiserFS v4

[Mike Fedyk]

On Fri, Sep 05, 2003 at 10:18:06PM +0200, Ragnar Kj?rstad wrote:
> On Fri, Sep 05, 2003 at 12:44:13PM -0700, Mike Fedyk wrote:
> > On Fri, Sep 05, 2003 at 01:15:41PM -0500, Grant Miner wrote:
> > > If you are making a packaging system, I recommend you do not allow 
> > > scipts, since they open up a whole can of worms.  (They're impossible to 
> > > invert, introduce security problems, make additional work for GUI 
> > > package managers, and cause too many problems.)
> > 
> > What do you replace them with then?
> 
> I think if scripts were to be removed from the packaging system you
> would have to add a package-feature for every single thing that scripts
> are currently used for. 
> 
> If you look through a collection of packages I think you will find that
> they typically do things like:
> - create users
> - start / stop deamons
> and so on.
> 
> So the package-system would need a way to specify that a user is part of
> the package (that would be very easy with a /etc/passwd.d-system) and
> that deamons that are part of the package should be started at install
> and stopped at uninstall-time.
> 
> There are probably dozens of other examples of common operations - and
> you would not be able to remove the script-feature unless 99% of what
> they are used for is not needed anymore - so it's a complicated task.

Great, so now you have a scripting language in the installer.  What did you
just achieve?

If you want gui integration of the scripts, just make nice wrappers of the
standard unix utilities that create a window saying "creating user x", or
prompting the user for data, or printing a notice.

You could probably even take the ncurses library, and add the wrappers there.

But let the people who want to do this choose the best way...

OT: package scripts (was Re: software packaging and ReiserFS v4)

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

2003-09-05T16:18:06 Ragnar Kj?rstad:
> I think if scripts were to be removed from the packaging system
> you would have to add a package-feature for every single thing
> that scripts are currently used for.

Ok, you've convinced me to this extent: when I get to a point where
I have to add such capabilities to my new packaging tool, they will
be added by the ability to invoke canned reusable scripts that are
part of the packaging system, not one-off per-package scripts.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: OT: package scripts (was Re: software packaging and ReiserFS v4)

[Grant Miner]

I'm excited about your packaging system.  I hope you make something very 
useful.  Here are some ideas I have had about packaging and operating 
system design; they might be interesting to you.  Comments, suggestion, 
other ideas are wanted.

Ideally, each user's home dir would mirror the system directories (i.e. 
users have their own bin, share, lib and var).  This way, users can 
install a package (and it would be local to them) and root could install 
the exact same package (and it would be global to the system).  Right 
now, on every distribution, only root can install packages; this is a 
place where a new packaging system could have huge advantages.

Ideally, users' directories would get unioned to system's directories 
(in their namespace) upon login, so / would be their "home", /bin would 
be system's bin plus user's bin, etc.  This way programs ALWAYS know the 
"right path" to everything (which has always been an annoying problem in 
Unix).

Packages contain file sets only.  This avoids side-effects of scripts. 
Well-designed programs do not need pre- and post-install scripts.  If 
you want to install apache, say as www user, root would create that 
user, switch users to www, and then install apache as www.

/sbin would go away; all sbin programs go to appropriate bin directories 
(because non-root users often need programs like fdisk, etc).

/etc, /bin, /lib and /boot all have the files only necessary for bootup.
Users can have an rc.d script directory in /etc (who gets to have rc.d 
scripts would be up to the sys admin) for launching services upon 
runlevel changes.  Perhaps it would be better to put these dirs as 
/boot/bin, /boot/lib, etc, and use /bin for system-wide bin (i.e., like 
the present /usr/bin).

/share/etc-defaults would contain default configurations, instead of 
cluttering up /etc (which is just stuff needed for booting and launching 
services).  /share/etc-defaults could be mounted below user's /etc.

(optional) /share/etc-mandatory could be mounted above user's /etc, for 
settings users should not be able to change (sometimes called "lock-down").

Environment variables are replaced by files, ala Plan 9.  Instead of 
$EDITOR have /etc/default-applications/text-editor, web-browser, etc. 
TIdeally no programs should use environment variables for configuration; 
their existence is not obvious to the user, and neither is their 
possible values. (Shell scripts of course would still use env vars.) 
Programs should watch their configuration files for changes, to achive 
that instant-apply effect.

I'm most excited for that once we have a stable Reiser4 system with 
files as dirs, it should be easy to write a universal configurator tool 
using files as keys.

Re: software packaging and ReiserFS v4

[Nikita Danilov]

Grant Miner writes:
 > If you are making a packaging system, I recommend you do not allow 
 > scipts, since they open up a whole can of worms.  (They're impossible to 
 > invert, introduce security problems, make additional work for GUI 
 > package managers, and cause too many problems.)

You one has support for the user-level transactions in the underlying
storage (_all_ storage), then scripts will at least be
"invertible". More or less so, of course, sending packages to the
network is hard to reverse.

 > 

Nikita.

Re: software packaging and ReiserFS v4

[Hans Reiser]

Nikita Danilov wrote:

>Grant Miner writes:
> > If you are making a packaging system, I recommend you do not allow 
> > scipts, since they open up a whole can of worms.  (They're impossible to 
> > invert, introduce security problems, make additional work for GUI 
> > package managers, and cause too many problems.)
>
>You one has support for the user-level transactions in the underlying
>storage (_all_ storage),
>
We don't yet support isolation, (but someday maybe we will have funding 
for it....)

> then scripts will at least be
>"invertible". More or less so, of course, sending packages to the
>network is hard to reverse.
>
> > 
>
>Nikita.
>
>
>  
>


-- 
Hans