Future of ext2 support in the Hurd?

Future of ext2 support in the Hurd?

[Theodore Ts'o]

Hi there,

There was recent discussion of taking out support for "dead" OS's and
features in e2fsprogs, such as fragmentation, support for the Masix
OS.... and Hurd.  So in the interests of doing some research to see
whether or not Hurd was really completely dead, or just "mostly dead"
(to in the Princess Bride sense :-), I did some digging, and found this
list.

There is an awful lot of out-of-date and extremely cobwebby pages out
there on the web, but it looks like even there are some people working
on it.  Am I right in my assumption that no one even though no one
touched the ext2 code in two years, with the bulk of the files in CVS
not having been touched in seven years, that there are still people
using the ext2 filesystem in Hurd?

One question which is going to be interesting from my perspective is
that of GPLv2 licensing.  There is definitely still code in the ext2
filesystem translator which is GPLv2 only, since it is derived from
Linux.  And as we all know, GPLv2 and GPLv3 code are licensing
incompatible, and that the FSF has claimed that the GPL will infect
across a wide variety of linking mechanisms, up to and including dynamic
linking.  Indeed, in the case of GCC, RMS has made the claim (when
pursuading NeXT to release the Objective C front-end under the GPL),
that the GPL infects across a Unix pipe!  The reason why I ask this
question is it seems extremely important whether or not the FSF has made
a determination if the GPL infects across HURD IPC calls.  If the GPLv2
does in fact infect across Hurd calls, and the Hurd is going GPLv3, it
seems that will be a need to either drop the ext2 filesystem, or rewrite
those portions of the ext2 filesystem which are derived from Linux code.

If the Hurd project is planning on dropping the ext2 filesystem, please
let me and the ext4 developers know, since then we can clean up the
special case code in e2fsprogs to support the Hurd.  

Thanks, regards,

						- Ted

Re: Future of ext2 support in the Hurd?

[Marcus Brinkmann]

Hi Theodore,

At Sun, 12 Aug 2007 17:40:00 -0400,
"Theodore Ts'o" <tytso@mit.edu> wrote:
> There was recent discussion of taking out support for "dead" OS's and
> features in e2fsprogs, such as fragmentation, support for the Masix
> OS.... and Hurd.  So in the interests of doing some research to see
> whether or not Hurd was really completely dead, or just "mostly dead"
> (to in the Princess Bride sense :-), I did some digging, and found this
> list.

Thanks for getting in contact with us!
 
> There is an awful lot of out-of-date and extremely cobwebby pages out
> there on the web, but it looks like even there are some people working
> on it.  Am I right in my assumption that no one even though no one
> touched the ext2 code in two years, with the bulk of the files in CVS
> not having been touched in seven years, that there are still people
> using the ext2 filesystem in Hurd?

Yes.  It's our primary filesystem, and if you pulled support for the
Hurd we would be forced to create and maintain a fork.

> One question which is going to be interesting from my perspective is
> that of GPLv2 licensing.

I will address this in a second message.

> If the Hurd project is planning on dropping the ext2 filesystem, please
> let me and the ext4 developers know, since then we can clean up the
> special case code in e2fsprogs to support the Hurd.  

We have no intention to drop support for it.  In fact, we are very
happy with it.  How much are we a burden for you?  If it helps, we can
probably arrange it so that a volunteer cooperates with you if work is
involved in keeping the support active.

Thanks,
Marcus

Re: Future of ext2 support in the Hurd?

[Theodore Tso]

On Mon, Aug 13, 2007 at 01:27:02AM +0200, Marcus Brinkmann wrote:
> We have no intention to drop support for it.  In fact, we are very
> happy with it.  How much are we a burden for you?  If it helps, we can
> probably arrange it so that a volunteer cooperates with you if work is
> involved in keeping the support active.

As long as there are people using it, we'll keep supporting it.  If we
could drop support for HURD-specific inode fields, it would clean up a
somewhat annoying union in the filesystem data structures, but that's
a minor point.

Something that *would* be good would be to have a volunteer
periodically check to make sure that the tip of the "maint" and
"master" branches of e2fsprogs builds and passes the regression test
suite on Hurd.  This may be somewhat happenning with the GNU/Hurd
Debian project, since the latest Debian version of e2fsprogs is
1.40.2, and I assume (although I don't know for sure) that the Debian
GNU/Hurd port is tracking Debian unstable, so if there were anything
embarassing, I would have gotten a complaint.  Still, it would be good
to explicitly run the regression test suite to make sure things are
working correctly --- and if not, to send patches.

This is just to make sure e2fsprogs is working fine for the Hurd, of
course; it's not like I would drop support if you can't find a
volunteer to do this, but at the moment we do no testing on Hurd and
there is no guarantees that a particular new release will work on the
Hurd because none of the active developers have a Hurd box.  

Regards,

					- Ted

Re: Future of ext2 support in the Hurd?

[Samuel Thibault]

Hi,

Theodore Tso, le Mon 13 Aug 2007 12:11:46 -0400, a écrit :
> the latest Debian version of e2fsprogs is 1.40.2, and I assume
> (although I don't know for sure) that the Debian GNU/Hurd port is
> tracking Debian unstable,

Yes.

> so if there were anything embarassing, I would have gotten a
> complaint.

I've just sent one on the bts :)
(just disabling devmapper, SElinux and online resizing support)

Samuel

Re: Future of ext2 support in the Hurd?

[Samuel Thibault]

Theodore Ts'o, le Sun 12 Aug 2007 17:40:00 -0400, a écrit :
> no one touched the ext2 code in two years,

Yes, because the current implementation roughly does its job, so I guess
nobody felt brave enough to touch it.  Support for >2GB filesystems was
developped and is used in Debian GNU/Hurd as a patch, though.

Regards,
Samuel

Re: Future of ext2 support in the Hurd?

[Marcus Brinkmann]

At Sun, 12 Aug 2007 17:40:00 -0400,
"Theodore Ts'o" <tytso@mit.edu> wrote:
> There is definitely still code in the ext2
> filesystem translator which is GPLv2 only, since it is derived from
> Linux.  And as we all know, GPLv2 and GPLv3 code are licensing
> incompatible, and that the FSF has claimed that the GPL will infect
> across a wide variety of linking mechanisms, up to and including dynamic
> linking.

Yes.  To my understanding, the GPL is not about technical methods, but
seeks for maximum protection available under law, irregardless of
technical methods to create a derived work.

The files you are referring to, ext2_fs.h and ext2_fs_i.h, can easily
be replaced by clean-room implementations, if relicensing the ext2fs
translator is desired.  However, we have other parts in the Hurd where
that is not possible (drivers in the kernel, network stack in our
network server).  So the questions you raise are still interesting.

In fact, I just searched for your name, and it pops up in the network
stack, but not in the ext2fs translator (the above files are copyright
Remy Card and Linus Torvald).

> Indeed, in the case of GCC, RMS has made the claim (when
> pursuading NeXT to release the Objective C front-end under the GPL),
> that the GPL infects across a Unix pipe!

I don't know any details about that case, but I can easily imagine
that being true, so I will just take your word for it.  The GPL FAQ
states that

  "We believe that a proper criterion depends both on the mechanism of
  communication (exec, pipes, rpc, function calls within a shared
  address space, etc.)  and the semantics of the communication (what
  kinds of information are interchanged).

  If the modules are included in the same executable file, they are
  definitely combined in one program. If modules are designed to run
  linked together in a shared address space, that almost surely means
  combining them into one program.

  By contrast, pipes, sockets and command-line arguments are
  communication mechanisms normally used between two separate
  programs. So when they are used for communication, the modules
  normally are separate programs. But if the semantics of the
  communication are intimate enough, exchanging complex internal data
  structures, that too could be a basis to consider the two parts as
  combined into a larger program."

This matches my best understanding of the issues involved.  Beyond
that, I can not say anything that applies particularly to the Hurd or
its RPC interfaces.

> The reason why I ask this
> question is it seems extremely important whether or not the FSF has made
> a determination if the GPL infects across HURD IPC calls.  If the GPLv2
> does in fact infect across Hurd calls, and the Hurd is going GPLv3, it
> seems that will be a need to either drop the ext2 filesystem, or rewrite
> those portions of the ext2 filesystem which are derived from Linux code.

That is an important question, but for now the Hurd is GPLv2, for
exactly that reason.  There are other significant parts of the Hurd
taken from Linux, so we can't do a complete switch at this time.  To
make a partial switch, we would have to address the issues you raised.

Beside the FSF' position, your position (and Remy Cards', Linus
Torvalds' etc) matters as well, of course.

As a side not, personally I am confident that most of these cases can
be resolved quickly and amicably, and will have exactly the outcome
that common sense suggests.  The other cases I will happily leave to
the lawyers to sort out.

Thanks,
Marcus

Re: Future of ext2 support in the Hurd?

[Theodore Tso]

On Mon, Aug 13, 2007 at 02:02:11AM +0200, Marcus Brinkmann wrote:
> In fact, I just searched for your name, and it pops up in the network
> stack, but not in the ext2fs translator (the above files are copyright
> Remy Card and Linus Torvald).

I only did a quick look, but I recognized some Linux code in balloc.c,
at least.  The code I think was probably written originally by Stephen
Tweedie.  We weren't very careful about maintaining authorship
attribution until after we moved to the BitKeeper (and later git) for
source code management.  I probably only have minimal kernel ownership
in terms of lines of code, with the exception being the directory
indexing changes and extended attribute code, which got reworked by me
before they were merged into mainline.  Most of my code ownership from
a copyright standpoint is in e2fsprogs.

> This matches my best understanding of the issues involved.  Beyond
> that, I can not say anything that applies particularly to the Hurd or
> its RPC interfaces.

It will be especially interesting since some RPC interfaces (mediated
by libc) will be used by GPLv2 userspace programs (such as e2fsck :-),
and some of those exact same interfaces will be used by Hurd
translators, and it will be interesting to see the legal
interpretations/contortions necessary to justify making some of these
distinctions.  :-)

> That is an important question, but for now the Hurd is GPLv2, for
> exactly that reason.  There are other significant parts of the Hurd
> taken from Linux, so we can't do a complete switch at this time.  To
> make a partial switch, we would have to address the issues you raised.
> 
> Beside the FSF' position, your position (and Remy Cards', Linus
> Torvalds' etc) matters as well, of course.

Well, the general consensus of the ext2/3/4 developers at this point
is to keep things at GPLv2.  My personal opinion matches with Linus
which is while I think that "Tivoization" in many cases will be a bad
choice from a marketing point of view (see how well Circuit City's
Divx venture fared), but it's not something that should be legislated
via the licenses.  If someone wants to use ext3 inside a locked down
device, that's fine with me, as long as I can get any improvements to
the source code back; I don't feel that it's worth it for the license
the mandate that I MUST be able to replace the running binaries inside
any embedded device.  So I don't see the Linux kernel going GPLv3 at
any time in the near future, and probably not ever.

Regards,

						- Ted

Re: Future of ext2 support in the Hurd?

[Thomas Bushnell BSG]

Currently the plan (approved by RMS) is to keep the Hurd libraries at
"GPLv2 or any later version" so that they can be linked with
Linux-derived code such as ext2fs stuff, and more importantly,
networking stuff.  The Hurd programs themselves (rather than just the
libraries) will be GPLv3, though the transition has not yet happened.

The particular programs that actually use GPLv2-only stuff (ext2fs,
pfinet) will remain as they are.

So thanks for the concern Ted!  It's an important issue, and one that we
think we have worked out correctly.

Thomas

Re: Future of ext2 support in the Hurd?

[Roland McGrath]

The license issues are not directly apropos to the main subject of
cooperation on supporting a particular disk format, so I'll leave that
aside.  Let us discuss the future of the EXT2_OS_HURD variant of ext2 format.

Indeed some people do use the Hurd and they all do rely on the EXT2_OS_HURD
format support in e2fsprogs.  It's the intended plan to migrate away from
EXT2_OS_HURD format and use a strict subset of the ext2 format features
used natively by Linux ext2fs (in theory eventually that whole set).  The
Hurd-specific data now stored in inode fields in EXT2_OS_HURD format would
instead be stored in ext2 xattr format.  The extent of cooperation then
required between ext2 format authorities and Hurd developers is assigning
an EXT2_XATTR_INDEX_* number to correspond to the "gnu." prefix.  The
conventions for formats and semantics of attributes with this prefix will
be maintained by the GNU Project.

Of course, I've been saying this was the plan for several years, and very
few steps of that migration have begun.  (You're probably not eager to see
my dusty patches for Linux ext2fs to support EXT2_OS_HURD format by mapping
its special data items to virtual xattrs with gnu.* names. :-) 

If it is important to you and you would like to state a quasi-formal
schedule to deprecate EXT2_OS_HURD format (not instantaneously), we can
work with that.  We'd hope for a decent interval of backward compatibility
support in e2fsck.  (The Hurd code will probably continue to support the
old format indefinitely.)  Hurd developers can make a small push to get our
ext2fs code supporting the xattr format encoding of Hurd-specific metadata.
If you're feeling especially charitable, you could provide format
conversion support in e2fsprogs.


Thanks,
Roland

Re: Future of ext2 support in the Hurd?

[Theodore Tso]

On Sun, Aug 12, 2007 at 08:13:36PM -0700, Roland McGrath wrote:
> Indeed some people do use the Hurd and they all do rely on the EXT2_OS_HURD
> format support in e2fsprogs.  It's the intended plan to migrate away from
> EXT2_OS_HURD format and use a strict subset of the ext2 format features
> used natively by Linux ext2fs (in theory eventually that whole set).  The
> Hurd-specific data now stored in inode fields in EXT2_OS_HURD format would
> instead be stored in ext2 xattr format.  The extent of cooperation then
> required between ext2 format authorities and Hurd developers is assigning
> an EXT2_XATTR_INDEX_* number to correspond to the "gnu." prefix.  The
> conventions for formats and semantics of attributes with this prefix will
> be maintained by the GNU Project.

Good plan.  :-)  Especially if you end up using a larger inode, and
supporting the EA-in-inode feature, I think you will find a huge speed
boost to files that use translators.  Out of curiosity, do you view
this as something which will be a common case, or an usual one (i.e.,
files using passive translators which are specified in the filesystem)?

Even if you don't, if a large number of files are always using the
same translator, there will probably be some significant disk space
savings with storing that information in an extended attribute.

> If it is important to you and you would like to state a quasi-formal
> schedule to deprecate EXT2_OS_HURD format (not instantaneously), we can
> work with that.  We'd hope for a decent interval of backward compatibility
> support in e2fsck.  (The Hurd code will probably continue to support the
> old format indefinitely.)  Hurd developers can make a small push to get our
> ext2fs code supporting the xattr format encoding of Hurd-specific metadata.
> If you're feeling especially charitable, you could provide format
> conversion support in e2fsprogs.

We don't need to give an explicit schedule, no.  I don't want to force
this to happen fast if the Hurd doesn't have volunteers lined up to do
the work.  As I mentioned earlier, the union is annoying, but it's
hardly a major obstacle to code maintainability.

I am certainly open to including format conversion in e2fprogs, if
someone sends me patches.  I doubt I will have the bandwidth to write
the patches myself in the near future; I have many other todo's on my
plate at the moment.  Working with you to assign a number for the name
index is something that I can help you do.  In fact, I think 7 is
available; why don't I reserve it and then get back to you once it's
been formally merged.

One thing that *would* be useful if you could provide some text (that
would go into the ext2/3/4 wiki) describe exactly what subset of the
ext2 filesystem is currently supported by the Hurd, and what the rough
roadmap you have for supporting additional features.  For example, do
you currently support sparse_superblocks, etc.  BTW, it may be that
you would want to supply a different /etc/mke2fs.conf file in the
e2fsprogs patches so that filesystems created by mke2fs don't include
features that might potentially give the Hurd ext2fs driver heartburn
--- I get the distinct feeling that it hasn't supported many of the
more recent innovations that are in the Linux implementation.

Regards,

						- Ted