Re: [Tux3] Current Activities?

Re: [Tux3] Current Activities?

[debian developer]

Hi Daniel,

On Fri, May 1, 2009 at 9:19 AM, Daniel Phillips<phillips@phunq.net> wrote:
> On Wednesday 29 April 2009, t3right.thebashar@xoxy.net wrote:
>> Hi Daniel,
>>
>> I want to apologize upfront if I sound like one of those "when will it
>> be done" questions that are best left unasked with most open source
>> projects.  Actually, I'm just really curious about what's been going
>> on lately.  I am somewhat embarrassed to admit that I became
>> fascinated with tux3's development since the first time it was
>> featured in an article on kerneltrap.  I've greatly enjoyed reading
>> your design notes and dialog with the btrfs team.  I had no naive
>> hopes that tux3 would get integrated into the linux kernel
>> immediately, but I've been extremely surprised at the loss of public
>> progress notes from you after the initial review request back and
>> forth died off.  In fact it seems like there's only been 7 postings
>> from you in the month since the last kernel merge related thread.
>>
>> I've really missed the public view into tux3's progress.  If it's not
>> too presumptuous, how are you? How's tux3 coming along?  What part of
>> the kernel port is taking the bulk of the work?  What new and
>> interesting challenges have you been wrestling with?
>
> Actually, I have been busy with real life for the last couple
> of months.  An interesting challenge is how I can keep up the
> previous development without any corporate support.  Zero.  Zip.
> Nada.  There has been exactly zero support from the usual
> suspects, who all have their own good reasons no doubt, which
> do not necessarily have much to do with the good of the Linux
> kernel or the Linux community.
>
> That challenge is being addressed.  Addressing it takes time and
> energy.  Development progress continues, though perhaps not as
> visible as it was earlier.  Being that visible takes a lot of time.
> The Tux3 watcher community could help a lot there.  For example,
> there is a large volume of technical material posted by me and
> others, that needs to be organized into a coherent form.  I simply
> don't have time to do that myself.  There are a bunch of projects
> like that.
>
> I also need more response to my occasional calls for volunteers.
> How many list readers have downloaded the code and run it?  You
> can in fact boot to a root fileystem with it.  How many besides
> me and Hirofumi have done that?
>
> Anyway, Andrew Morton was right, we should have merged into
> mainline as soon as Tux3 was booting as root.  That would have
> taken a big load off me.  Instead, somebody posted to LKML and
> called for atomic commit as a precondition for merging.  Sounds
> like a good idea, sounds logical.  But actually, in open source
> it is counter productive, it just puts a bigger load on me, a
> limited resource.  We should have merged first, then got the
> logging and replay working.  In fact, we probably should still do
> that.  I will say this now: if we are invited to merge in the
> next major release, or in -mm or whatever, we will happily do it.
> If we are not invited to merge, nobody has any cause to complain
> about progress slowing down.
>
> Anyway, now would be a good time to have a discussion here on the
> Tux3 list about what can be done to get more helping hands
> involved in the heavy lifting.
>

A list of beginner projects with instructions on how to get off would
be helpful.
Beginners need to be motivated to hang around long enough to become core
developers. I am sure you will not find many core developers who are willing to
find time to hack on another project, almost everyone has their hands full.

Other thing is the need to talk to organizations. You could have
easily listed tux3
for Google Summer of Code 2009. It would have brought attention and resources
for the project. I am sure you will find financing as you are well
recognized in your
domain. I am including LKML so that others will hopefully notice and respond.

May be jonathan can help in advertising the need for developers better
through LWN.

This can be easily done by one of us if there is a wiki. Also a wiki helps us to
collate things which we have learned till now.


So two things:

1. Put up a wiki (important)
2. List of starter projects.

I see the dedup feature is partially complete. May be it can be completed and
included in the kernel source of tux3. We need more projects like this.

Hope to see some activity soon. Do not let the project die.

Thanks,

Re: [Tux3] Current Activities?

[Daniel Phillips]

On Friday 31 July 2009, debian developer wrote:
> Hi Daniel,
> 
> On Fri, May 1, 2009 at 9:19 AM, Daniel Phillips<phillips@phunq.net> wrote:
> > On Wednesday 29 April 2009, t3right.thebashar@xoxy.net wrote:
> >> Hi Daniel,
> >>
> >> I want to apologize upfront if I sound like one of those "when will it
> >> be done" questions that are best left unasked with most open source
> >> projects.  Actually, I'm just really curious about what's been going
> >> on lately.  I am somewhat embarrassed to admit that I became
> >> fascinated with tux3's development since the first time it was
> >> featured in an article on kerneltrap.  I've greatly enjoyed reading
> >> your design notes and dialog with the btrfs team.  I had no naive
> >> hopes that tux3 would get integrated into the linux kernel
> >> immediately, but I've been extremely surprised at the loss of public
> >> progress notes from you after the initial review request back and
> >> forth died off.  In fact it seems like there's only been 7 postings
> >> from you in the month since the last kernel merge related thread.
> >>
> >> I've really missed the public view into tux3's progress.  If it's not
> >> too presumptuous, how are you? How's tux3 coming along?  What part of
> >> the kernel port is taking the bulk of the work?  What new and
> >> interesting challenges have you been wrestling with?
> >
> > Actually, I have been busy with real life for the last couple
> > of months.  An interesting challenge is how I can keep up the
> > previous development without any corporate support.  Zero.  Zip.
> > Nada.  There has been exactly zero support from the usual
> > suspects, who all have their own good reasons no doubt, which
> > do not necessarily have much to do with the good of the Linux
> > kernel or the Linux community.
> >
> > That challenge is being addressed.  Addressing it takes time and
> > energy.  Development progress continues, though perhaps not as
> > visible as it was earlier.  Being that visible takes a lot of time.
> > The Tux3 watcher community could help a lot there.  For example,
> > there is a large volume of technical material posted by me and
> > others, that needs to be organized into a coherent form.  I simply
> > don't have time to do that myself.  There are a bunch of projects
> > like that.
> >
> > I also need more response to my occasional calls for volunteers.
> > How many list readers have downloaded the code and run it?  You
> > can in fact boot to a root fileystem with it.  How many besides
> > me and Hirofumi have done that?
> >
> > Anyway, Andrew Morton was right, we should have merged into
> > mainline as soon as Tux3 was booting as root.  That would have
> > taken a big load off me.  Instead, somebody posted to LKML and
> > called for atomic commit as a precondition for merging.  Sounds
> > like a good idea, sounds logical.  But actually, in open source
> > it is counter productive, it just puts a bigger load on me, a
> > limited resource.  We should have merged first, then got the
> > logging and replay working.  In fact, we probably should still do
> > that.  I will say this now: if we are invited to merge in the
> > next major release, or in -mm or whatever, we will happily do it.
> > If we are not invited to merge, nobody has any cause to complain
> > about progress slowing down.
> >
> > Anyway, now would be a good time to have a discussion here on the
> > Tux3 list about what can be done to get more helping hands
> > involved in the heavy lifting.
> >
> 
> A list of beginner projects with instructions on how to get off would
> be helpful.
> Beginners need to be motivated to hang around long enough to become core
> developers. I am sure you will not find many core developers who are willing to
> find time to hack on another project, almost everyone has their hands full.

For your information, I have my hands more than full with things entirely
unrelated to filesystem hacking.  This situation will last another few
days then I should be able to refocus at least part of my time to the
project.  Which I still believe in completely, and want to have running
on my own computers as soon as possible.

Anyway, let me offer two beginner projects:

  1) Add success checks to unit tests
  2) add a "check" command to the tux3 command

Unit test success:

  - Unit tests are compiled as main programs in tux/user, that include
    kernel files and sometimes other userspace files as source text,
    for example: #include "kernel/filemap.c"

  - Currently, unit tests just run and produce output.  If something is
    wrong an assertion may trigger, or it may segfault, or the tracing
    output may be unreasonable.

  - It would be better if the test concluded with a test to see that
    then final result is as expected, according to the operations that
    were performed.  The main program should return 0 with no warning
    if results are correct, otherwise print a warning and return 1, a
    failure code for a shell script we may add later.

Tux3 check:

   - Command form should be:

       tux3 check <volume> [<file>]

   - As a first approximation, should just walk the filesystem tree from
     the root as Hirofumi's tux3graph program does, checking that
     physical block pointers lie within the volume and that leaf nodes
     of the inode table and file data subtrees pass the leaf checks
     already implemented.

   - Later, this walk will build a bitmap map of used blocks so that we
     can check that there is no lost space in the volume and that actual
     used/free matches the allocation bitmaps.  When building the bitmap
     we can also check for multiple pointers to the same block, which
     are not allowed in tux3.

> Other thing is the need to talk to organizations. You could have
> easily listed tux3

Anybody who wants to do that is welcome to, anybody who wants to can
appoint themselves an official representative of the tux3 project, there
is no need to ask.  Just come by irc and let us know what happened :-)

> for Google Summer of Code 2009. It would have brought attention and resources
> for the project. I am sure you will find financing as you are well
> recognized in your
> domain. I am including LKML so that others will hopefully notice and respond.
> 
> May be jonathan can help in advertising the need for developers better
> through LWN.
> 
> This can be easily done by one of us if there is a wiki. Also a wiki helps us to
> collate things which we have learned till now.

Shapor maintains our website, which does have a wiki.  Just ping him on
irc about it.

> So two things:
> 
> 1. Put up a wiki (important)
> 2. List of starter projects.
> 
> I see the dedup feature is partially complete. May be it can be completed and
> included in the kernel source of tux3. We need more projects like this.
> 
> Hope to see some activity soon. Do not let the project die.

As you just demonstrated, it can't die, it's open source.

Regards,

Daniel

Re: [Tux3] Current Activities?

[Theodore Tso]

On Sat, Aug 01, 2009 at 09:42:51AM -0700, Daniel Phillips wrote:
> Shapor maintains our website, which does have a wiki.  Just ping him on
> irc about it.

Free hint: make a pointer to the wiki be much more prominent on the
tux3.org web site.  I tried looking for it on the tux3 web site, and
then on google, and the closest thing I could find was this empty wiki:

     http://bitbucket.org/shapor/tux3/wiki/Home

One thing which you might want to include is what makes the tux3 file
system unique and why someone might want to use tux3 instead of some
other file system?  The web site talks about "versioned pointer" which
is a technical feature that might appeal to a computer scientist ---
but what does that mean to an end user?

Performance?  Features?  If so, which features?  From what I can tell,
the feature set of tux3 seems to be a subset of btrfs; is there some
feature or features that tux3 have that other filesystems will not?
Not that I advocate file system developers throwing elbows or
otherwise being too competitive with each other, but the reality is
that every file system design makes tradeoffs, giving it certain
strengths, and some weaknesses.  So what *is* tux3's core design
focus, in terms of workload, or user base?  Making a clear statement
about what the goals of the tux3 filesystem on the web site is
certainly one of those things that I would recommend.

Best regards,

					- Ted

Re: [Tux3] Current Activities?

[Ingo Molnar]

* Daniel Phillips <phillips@phunq.net> wrote:

> [...]  I will say this now: if we are invited to merge in the next 
> major release, or in -mm or whatever, we will happily do it. If we 
> are not invited to merge, nobody has any cause to complain about 
> progress slowing down. [...]

The thing is, if you are waiting for an 'invite to upstream Linux 
merge' that could be a _very_ long wait: i've yet to see a single 
one ever since i started hacking Linux ~14 years ago ;-)

The model that Linux has been following for the past 10+ years is 
for new kernel projects to request inclusion. I.e. you push your 
patches upstream: you send patches and a pull request to the 
appropriate people/lists such as lkml.

This is done so because merging patches is a fundamentally 
hierarchical process, and the people merging _your_ patches are the 
real maintenance bottleneck, not you.

So it is not Linus and other maintainers who are searching the web 
for projects to merge and sending out 'invites' but the other way 
around: projects try to get upstream by submitting patches (which 
get reviewed and accepted or rejected).

So if you'd like your code to be merged upstream you better start 
this process now - this alone can take a lot of time: months (or 
years in certain cases). But it is still a much shorter time-span 
than an 'invite to merge' ;-)

	Ingo

Re: [Tux3] Current Activities?

[Daniel Phillips]

On Thursday 06 August 2009, Ingo Molnar wrote:
> 
> * Daniel Phillips <phillips@phunq.net> wrote:
> 
> > [...]  I will say this now: if we are invited to merge in the next 
> > major release, or in -mm or whatever, we will happily do it. If we 
> > are not invited to merge, nobody has any cause to complain about 
> > progress slowing down. [...]
> 
> The thing is, if you are waiting for an 'invite to upstream Linux 
> merge' that could be a _very_ long wait: i've yet to see a single 
> one ever since i started hacking Linux ~14 years ago ;-)

Hi Ingo,

We were already sorta-kinda invited to merge, as I read it, having received advice to merge early while the code base remains small
instead of adding more features and risking premature bloat.  I seem to
recall there was a dissenter at that time on the basis of lack of
atomic commit, and we chose to follow the latter advice instead of the
former, arguably a mistake.

> The model that Linux has been following for the past 10+ years is 
> for new kernel projects to request inclusion. I.e. you push your 
> patches upstream: you send patches and a pull request to the 
> appropriate people/lists such as lkml.
> 
> This is done so because merging patches is a fundamentally 
> hierarchical process, and the people merging _your_ patches are the 
> real maintenance bottleneck, not you.
> 
> So it is not Linus and other maintainers who are searching the web 
> for projects to merge and sending out 'invites' but the other way 
> around: projects try to get upstream by submitting patches (which 
> get reviewed and accepted or rejected).

With 20-20 hindsight, we should have gone for a merge immediately after
the SCALE 7X demo, at which time Tux3 was reliably but without
atomic commit.  Instead, we decided to heed those asking for atomic
commit before merge.  This turned out to be a bad idea, because my
available time soon became severely constrained.  Work on atomic commit
has progressed over the last few months, but slowly.  If we had merged
back in February this work would have progressed faster and been done
by now.  As it is, we will probably forge on with the atomic commit work
rather than changing course and looking for a merge right away, because
there has been considerable progress in that direction.  Also, my time
is not quite as constrained as it was - still constrained, but not quite
as severely as recently.

Meanwhile Hirofumi has dauntlessly forged on, extracting design details
from me, adding some of his own, generally subjecting everything to
minute scrutiny, and in the process turning out a significant volume of
high quality code.  If I had to state what the best thing that could
happen to Tux3 is, it would be: somebody steps up to sponsor Hirofumi.
He has done all that work strictly as a volunteer without support of any
kind, and he could use some.  Not to mention the fact that as FAT
maintainer we would be well served to have Hirofumi in a comfort zone,
given recent developments on that front.

> So if you'd like your code to be merged upstream you better start 
> this process now - this alone can take a lot of time: months (or 
> years in certain cases). But it is still a much shorter time-span 
> than an 'invite to merge' ;-)
> 
> 	Ingo

My "invite to merge" comment was made in the context of various comments
about insufficiently rapid progress, and should not be taken out of
context.  We are well aware of the traditional merging procedure.  We
certainly are not waiting for any maintainer action, but rather for more
progress from ourselves.  There are various possibilities to speed that
up.  See "sponsor Hirofumi" above.  I myself do not need sponsorship at
this point, what I need is more time on my hands, and only I can do
anything about that.  (As of today, I do have a little more time
available, hence this post.)

In fact, our merge process has been in progress since February.  As we
see it, it goes like this:

  1) Implement and verify atomic commit correctness at least for a
     large subset of filesystem operations.

  2) Clean the kernel part of the code base to a respectable state.

  3) Commence our post-address-repost cycle with intent to merge.

We are at step 1 and I can't say how long it will take to get to step 2.
It is on the way.  Any help we get will hurry it along.

I will very briefly address Ted's question, which amounts to: "why
do we want Tux3?"  I think Tux3 fills an empty niche in our filesystem
ecology where a simple, clean and modern general purpose filesystem
should exist and there is none.  In concrete terms, Tux3 implements a
single-pointer-per-extent model that Btrfs and ZFS do not.  This allows
a very simple *physical* design, with much complexity pushed to the
*logical* level where things generally behave better.  A simple physical
design offers many benefits, including making it easier to take a run at
that holiest of holy grails, online check and repair.

In even more concrete terms, Tux3 already demonstrates a tiny memory
footprint, unlike Btrfs or ZFS.  Its CPU footprint is also tiny.  As
such, Tux3 could easily run on a cellphone or smaller device, while
offering a full range of "big filesystem" capabilities.  (Note that
Zumastor already proves we know how to do replication properly:
http://zumastor.org/.)

Regards,

Daniel

Re: [Tux3] Current Activities?

[debian developer]

Hi Daniel,

On Sun, Aug 9, 2009 at 5:17 AM, Daniel Phillips<phillips@phunq.net> wrote:

>
> Meanwhile Hirofumi has dauntlessly forged on, extracting design details
> from me, adding some of his own, generally subjecting everything to
> minute scrutiny, and in the process turning out a significant volume of
> high quality code.  If I had to state what the best thing that could
> happen to Tux3 is, it would be: somebody steps up to sponsor Hirofumi.
> He has done all that work strictly as a volunteer without support of any
> kind, and he could use some.  Not to mention the fact that as FAT
> maintainer we would be well served to have Hirofumi in a comfort zone,
> given recent developments on that front.
>

The best thing to do until Hirofumi gets sponsorship is to sponsor him by
donations. I think it is easy enough for you to set up a PayPal donation
link on the tux3 homepage. I hope these small contributions will help you
to sponsor him until you get the corporate sponsorship. I, for one would be
willing to donate my small part if it helps in the development of Tux3.

Thanks,
DD

Re: [Tux3] Current Activities?

[Theodore Tso]

On Sat, Aug 08, 2009 at 04:47:21PM -0700, Daniel Phillips wrote:
> I will very briefly address Ted's question, which amounts to: "why
> do we want Tux3?"  I think Tux3 fills an empty niche in our filesystem
> ecology where a simple, clean and modern general purpose filesystem
> should exist and there is none.  

You'll need to define "simple", "clean", and "modern" here; but this
isn't something that makes a lot of sense to an end-user.

> In concrete terms, Tux3 implements a
> single-pointer-per-extent model that Btrfs and ZFS do not.  This allows
> a very simple *physical* design, with much complexity pushed to the
> *logical* level where things generally behave better.  

This isn't very compelling, either....

> A simple physical
> design offers many benefits, including making it easier to take a run at
> that holiest of holy grails, online check and repair.

Note that a simple physical design does not necessarily enable online
check/repair.  In fact, without back pointers (which add complexity),
doing online checking can be completely impractical for large
filesystems.  There's been quite a lot of thinking on this subject;
see Val's "Repair-driven Filesystem Design" paper.

> In even more concrete terms, Tux3 already demonstrates a tiny memory
> footprint, unlike Btrfs or ZFS.  Its CPU footprint is also tiny.  As
> such, Tux3 could easily run on a cellphone or smaller device, while
> offering a full range of "big filesystem" capabilities.  (Note that
> Zumastor already proves we know how to do replication properly:
> http://zumastor.org/.)

Well, btrfs currently weighs in at 302k of compiled object code.
While that's certainly not small (ext4+jbd2 weighs in at 256k;
ext3+jbd weighs in at 124k; ext2 at 42k), 300k isn't going to break
the bank on most cell phones that would be running Linux these days.
And on a cell phone, it might very well be that some flash-based
filesystem such as JFFS2 or UBIFS would be a better choice in any
case.

If the goal is to find corporate sponsorship for Tux3, I'd strongly
encourage you to think harder about a compelling story for why Tux3 is
so cool that companies should spend money supporting it.  Let me
gently suggest to you that "it'll have fewer features than btrfs, but
it will use less memory" is not a particularly compelling story to a
company's technical and management leadership who is figure out
spending priorities for next year's budget.  Particularly if the cell
phone is going to have megabytes of memory to run Java on it anyway;
and even if it's not running Java, have you seen how much space
graphical libraries take up these days?  :-)

Again, I'm not saying this to discourage technical people from working
on Tux3.  But just because you're passionate about a technology,
doesn't mean that it automatically translate to there being a business
case to convince companies to invest in that technology.

Best regards,

              	   	    	       	   	- Ted

Re: [Tux3] Current Activities?

[OGAWA Hirofumi]

Theodore Tso <tytso@mit.edu> writes:

> If the goal is to find corporate sponsorship for Tux3, I'd strongly
> encourage you to think harder about a compelling story for why Tux3 is
> so cool that companies should spend money supporting it.  Let me
> gently suggest to you that "it'll have fewer features than btrfs, but
> it will use less memory" is not a particularly compelling story to a
> company's technical and management leadership who is figure out
> spending priorities for next year's budget.  Particularly if the cell
> phone is going to have megabytes of memory to run Java on it anyway;
> and even if it's not running Java, have you seen how much space
> graphical libraries take up these days?  :-)
>
> Again, I'm not saying this to discourage technical people from working
> on Tux3.  But just because you're passionate about a technology,
> doesn't mean that it automatically translate to there being a business
> case to convince companies to invest in that technology.

About sponsorship, I guess Daniel just worried about me. But, it's not
argument on lkml. So, let's stop argument about sponsorship.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: [Tux3] Current Activities?

[david]

On Mon, 10 Aug 2009, OGAWA Hirofumi wrote:

> Theodore Tso <tytso@mit.edu> writes:
>
>> If the goal is to find corporate sponsorship for Tux3, I'd strongly
>> encourage you to think harder about a compelling story for why Tux3 is
>> so cool that companies should spend money supporting it.  Let me
>> gently suggest to you that "it'll have fewer features than btrfs, but
>> it will use less memory" is not a particularly compelling story to a
>> company's technical and management leadership who is figure out
>> spending priorities for next year's budget.  Particularly if the cell
>> phone is going to have megabytes of memory to run Java on it anyway;
>> and even if it's not running Java, have you seen how much space
>> graphical libraries take up these days?  :-)
>>
>> Again, I'm not saying this to discourage technical people from working
>> on Tux3.  But just because you're passionate about a technology,
>> doesn't mean that it automatically translate to there being a business
>> case to convince companies to invest in that technology.
>
> About sponsorship, I guess Daniel just worried about me. But, it's not
> argument on lkml. So, let's stop argument about sponsorship.

it's not just sponsership, it's also the question of why should I as 
someone who uses linux in production ever care about Tux3. why would I 
pick it over ext4, btrfs, etc.

David Lang