Re: [RFC] Btrfs mainline plans

Re: [RFC] Btrfs mainline plans

[Adrian Bunk]

On Sun, Oct 05, 2008 at 09:11:13AM -0500, Serge E. Hallyn wrote:
> Quoting Adrian Bunk (bunk@kernel.org):
> > On Fri, Oct 03, 2008 at 12:18:59AM -0700, Andrew Morton wrote:
> > > On Mon, 29 Sep 2008 15:44:20 -0400 Chris Mason <chris.mason@oracle.com> wrote:
> > > 
> > > > But, the code is very actively developed, and I believe the best way to
> > > > develop Btrfs from here is to get it into the mainline kernel (with a
> > > > large warning label about the disk format) and attract more extensive
> > > > review of both the disk format and underlying code.
> > > 
> > > For the record...  I have been encouraging Chris to get btrfs into
> > > mainline soon.  Get it into linux-next asap and merge it into 2.6.29.
> > > 
> > > And do this even though the on-disk format is still changing - we emit a
> > > loud printk at mount time and if someone comes to depend upon some
> > > intermediate format, well, that's their tough luck.
> > > 
> > > My thinking here is that btrfs probably has a future, and that an early
> > > merge will accelerate its development and will broaden its developer base. 
> > > If it ends up failing for some reason, well, we can just delete it
> > > again.
> > > 
> > > For various reasons this approach often isn't appropriate as a general
> > > policy thing, but I do think that Linux has needed a new local
> > > filesystem for some time, and btrfs might be The One, and hence is
> > > worth a bit of special-case treatment.
> > 
> > Let's try to learn from the past:
> > 
> > 6 days from today ext4 (another new local filesystem for Linux) 
> > celebrates the second birthday of it's inclusion into Linus' tree
> > as a similar special-case.
> > 
> > You claim "an early merge will accelerate its development and will 
> > broaden its developer base" for Btrfs.
> > 
> > Read the timeline Ted outlined back in June 2006 for ext4 [1].
> > When comparing with what happened in reality it kinda disproves
> > your "acceleration" point.
> 
> OTOH, maybe it's just me, but I think there is more excitement around
> btrfs.  Myself I'm dying for snapshot support, and can't wait to try
> btrfs on a separate data/scratch partition (where i don't mind losing
> data).  btrfs and nilfs - yay.  Ext4?  <yawn>  That can make all the
> difference.

"accelerate its development and will broaden its developer base" is not 
about users/testers but about people doing code development.

For people wanting to try WIP code you don't need it in mainline.

Stable kernels will anyway usually contain months old code of the
WIP filesystem that is not usable for testing, so for any meaningful
testing you will still have to follow the btrfs tree and not mainline.

This is not meant as a statement on the quality of ext4 or btrfs, or any 
comparison of the development times of ext4 and btrfs, but for ext4 the 
advantages Andrew thinks would happen with an early btrfs merge do not 
seem to have happened.

I just realize that I forgot to add Ted and the ext4 mailing list into 
the Cc of my first email. Adding them to the Cc, so if I'm talking 
nonsense about the experiences with ext4 they can correct me.

> -serge

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [RFC] Btrfs mainline plans

[Chris Mason]

On Sun, 2008-10-05 at 18:09 +0300, Adrian Bunk wrote:
> On Sun, Oct 05, 2008 at 09:11:13AM -0500, Serge E. Hallyn wrote:
> > Quoting Adrian Bunk (bunk@kernel.org):
> >

[ when to merge btrfs ]

> > > Let's try to learn from the past:
> > > 
> > > 6 days from today ext4 (another new local filesystem for Linux) 
> > > celebrates the second birthday of it's inclusion into Linus' tree
> > > as a similar special-case.
> > > 
> > > You claim "an early merge will accelerate its development and will 
> > > broaden its developer base" for Btrfs.
> > > 
> > > Read the timeline Ted outlined back in June 2006 for ext4 [1].
> > > When comparing with what happened in reality it kinda disproves
> > > your "acceleration" point.
> > 

The btrfs timelines have always been aggressive, and as btrfs gets
closer to feature complete, the testing matrix grows dramatically.  I
can't promise my crazy timelines won't slip, but I've been hacking away
in the basement for almost 18 months now and it's time for me to get off
the pot and make it stable.

Ext4 has always had to deal with the ghost of ext3.  Both from a
compatibility point of view and everyone's expectations of stability.  I
believe that most of us underestimated how difficult it would be to move
ext4 forward.

Btrfs is different for lots of reasons, and being in mainline will
definitely increase the pressure on the btrfs developers to finish, and
the resources available for us to finish with.

> > OTOH, maybe it's just me, but I think there is more excitement around
> > btrfs.  Myself I'm dying for snapshot support, and can't wait to try
> > btrfs on a separate data/scratch partition (where i don't mind losing
> > data).  btrfs and nilfs - yay.  Ext4?  <yawn>  That can make all the
> > difference.
> 
> "accelerate its development and will broaden its developer base" is not 
> about users/testers but about people doing code development.
> 

People want btrfs for different reasons.  I want btrfs in the kernel
because when you're in the kernel more people look at it, and when
people look at it they send me email with the mistakes they found.

For example, see the streaming write patches I sent to fsdevel last
week.  I wouldn't test against ext4 as often if I had to hunt down
external repos just to get something consistent with the current
development kernels.  ext4 in mainline makes it much easier for me to
kick the tires.

> For people wanting to try WIP code you don't need it in mainline.
> 
> Stable kernels will anyway usually contain months old code of the
> WIP filesystem that is not usable for testing, so for any meaningful
> testing you will still have to follow the btrfs tree and not mainline.

For ext4 at least, the mainline code is very usable.  I hope to have
btrfs in shape for that by the 2.6.29 merge cycle.

-chris

Re: [RFC] Btrfs mainline plans

[Adrian Bunk]

On Mon, Oct 06, 2008 at 09:40:03AM -0400, Chris Mason wrote:
> On Sun, 2008-10-05 at 18:09 +0300, Adrian Bunk wrote:
> > On Sun, Oct 05, 2008 at 09:11:13AM -0500, Serge E. Hallyn wrote:
> > > Quoting Adrian Bunk (bunk@kernel.org):
> > >
> 
> [ when to merge btrfs ]
> 
> > > > Let's try to learn from the past:
> > > > 
> > > > 6 days from today ext4 (another new local filesystem for Linux) 
> > > > celebrates the second birthday of it's inclusion into Linus' tree
> > > > as a similar special-case.
> > > > 
> > > > You claim "an early merge will accelerate its development and will 
> > > > broaden its developer base" for Btrfs.
> > > > 
> > > > Read the timeline Ted outlined back in June 2006 for ext4 [1].
> > > > When comparing with what happened in reality it kinda disproves
> > > > your "acceleration" point.
> > > 
> 
> The btrfs timelines have always been aggressive, and as btrfs gets
> closer to feature complete, the testing matrix grows dramatically.  I
> can't promise my crazy timelines won't slip, but I've been hacking away
> in the basement for almost 18 months now and it's time for me to get off
> the pot and make it stable.
> 
> Ext4 has always had to deal with the ghost of ext3.  Both from a
> compatibility point of view and everyone's expectations of stability.  I
> believe that most of us underestimated how difficult it would be to move
> ext4 forward.
> 
> Btrfs is different for lots of reasons, and being in mainline will
> definitely increase the pressure on the btrfs developers to finish, and
> the resources available for us to finish with.

Your last sentence does not make sense:

According to your timeline btrfs 1.0 will be released in Q408 [1] - and
the merge window for 2.6.29 will be in Q109.

>...
> > For people wanting to try WIP code you don't need it in mainline.
> > 
> > Stable kernels will anyway usually contain months old code of the
> > WIP filesystem that is not usable for testing, so for any meaningful
> > testing you will still have to follow the btrfs tree and not mainline.
> 
> For ext4 at least, the mainline code is very usable.  I hope to have
> btrfs in shape for that by the 2.6.29 merge cycle.

One risk you should be aware of is that when btrfs is in 2.6.29 part of 
the Linux press might pick it up and stress test and benchmark this new 
filesystem.

JFS still suffers from from not being that good when it was
initially merged.

> -chris

cu
Adrian

[1] http://btrfs.wiki.kernel.org/index.php/Development_timeline

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [RFC] Btrfs mainline plans

[Chris Mason]

On Tue, 2008-10-07 at 18:27 +0300, Adrian Bunk wrote:
> On Mon, Oct 06, 2008 at 09:40:03AM -0400, Chris Mason wrote:
>  
> > 
> > The btrfs timelines have always been aggressive, and as btrfs gets
> > closer to feature complete, the testing matrix grows dramatically.  I
> > can't promise my crazy timelines won't slip, but I've been hacking away
> > in the basement for almost 18 months now and it's time for me to get off
> > the pot and make it stable.
> > 
> > Ext4 has always had to deal with the ghost of ext3.  Both from a
> > compatibility point of view and everyone's expectations of stability.  I
> > believe that most of us underestimated how difficult it would be to move
> > ext4 forward.
> > 
> > Btrfs is different for lots of reasons, and being in mainline will
> > definitely increase the pressure on the btrfs developers to finish, and
> > the resources available for us to finish with.
> 
> Your last sentence does not make sense:
> 
> According to your timeline btrfs 1.0 will be released in Q408 [1] - and
> the merge window for 2.6.29 will be in Q109.
> 

Planning for mainline inclusion is always a guessing game.  Cutting 1.0
is different from being in mainline, and the dates don't have to be the
same.

> >...
> > > For people wanting to try WIP code you don't need it in mainline.
> > > 
> > > Stable kernels will anyway usually contain months old code of the
> > > WIP filesystem that is not usable for testing, so for any meaningful
> > > testing you will still have to follow the btrfs tree and not mainline.
> > 
> > For ext4 at least, the mainline code is very usable.  I hope to have
> > btrfs in shape for that by the 2.6.29 merge cycle.
> 
> One risk you should be aware of is that when btrfs is in 2.6.29 part of 
> the Linux press might pick it up and stress test and benchmark this new 
> filesystem.

I think the gains from early testing far outweigh the risks of bad early
press.

-chris

Re: [RFC] Btrfs mainline plans

[Adrian Bunk]

On Tue, Oct 07, 2008 at 12:01:24PM -0400, Chris Mason wrote:
> On Tue, 2008-10-07 at 18:27 +0300, Adrian Bunk wrote:
> > On Mon, Oct 06, 2008 at 09:40:03AM -0400, Chris Mason wrote:
> >  
> > > 
> > > The btrfs timelines have always been aggressive, and as btrfs gets
> > > closer to feature complete, the testing matrix grows dramatically.  I
> > > can't promise my crazy timelines won't slip, but I've been hacking away
> > > in the basement for almost 18 months now and it's time for me to get off
> > > the pot and make it stable.
> > > 
> > > Ext4 has always had to deal with the ghost of ext3.  Both from a
> > > compatibility point of view and everyone's expectations of stability.  I
> > > believe that most of us underestimated how difficult it would be to move
> > > ext4 forward.
> > > 
> > > Btrfs is different for lots of reasons, and being in mainline will
> > > definitely increase the pressure on the btrfs developers to finish, and
> > > the resources available for us to finish with.
> > 
> > Your last sentence does not make sense:
> > 
> > According to your timeline btrfs 1.0 will be released in Q408 [1] - and
> > the merge window for 2.6.29 will be in Q109.
> 
> Planning for mainline inclusion is always a guessing game.

The 2.6.29 merge window will start in January - everything else is much
more unlikely than an incompetent chick from Alaska getting only one 
heart attack away from the big red button.

> Cutting 1.0
> is different from being in mainline, and the dates don't have to be the
> same.

Sure, but Andrew's "special-case treatment" suggestion does not apply if 
1.0 gets released before the 2.6.29 merge window.

>...
> -chris

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [RFC] Btrfs mainline plans

[Daniel Phillips]

On Sunday 05 October 2008 08:09, Adrian Bunk wrote:
> "accelerate its development and will broaden its developer base" is not 
> about users/testers but about people doing code development.
> 
> For people wanting to try WIP code you don't need it in mainline.

But it saves time for the user, who does not have to run around chasing
links, carefully checking for a kernel match, downloading, patching,
building and installing a single purpose kernel, and bringing it up on
a machine that would probably have only required one click on the new
filesystem option otherwise.  The considerable time thus saved can be
invested profitably in running test cases and filing bug reports.
 
> Stable kernels will anyway usually contain months old code of the
> WIP filesystem that is not usable for testing, so for any meaningful
> testing you will still have to follow the btrfs tree and not mainline.

True, but the trick here is getting started.  It is much easier to
justify the effort of going out and getting the latest patch if one
knows from experience that it basically already works.

> This is not meant as a statement on the quality of ext4 or btrfs, or any 
> comparison of the development times of ext4 and btrfs, but for ext4 the 
> advantages Andrew thinks would happen with an early btrfs merge do not 
> seem to have happened.

Are you sure about that?  I see 33 messages on linux-ext4 yesterday,
from a broad range of contributors.  Versus eight from a much narrower
range of contributors, Oct 4 a year ago.

There is little question that an early merge helps both developers and
users employ their time more efficiently, once a project is past the
point where we wonder about its value and/or viability.  In my opinion,
Btrfs clearly has both.  Particularly because we need a way to stem the
loss of mindshare to ZFS in the storage space, which is significant at
the moment.  And Btrfs is closest to the finish line in that regard.
It needs all the help it can get.

Regards,

Daniel

Re: [RFC] Btrfs mainline plans

[Adrian Bunk]

On Wed, Oct 08, 2008 at 02:33:32PM -0700, Daniel Phillips wrote:
> On Sunday 05 October 2008 08:09, Adrian Bunk wrote:
> > "accelerate its development and will broaden its developer base" is not 
> > about users/testers but about people doing code development.
> > 
> > For people wanting to try WIP code you don't need it in mainline.
> 
> But it saves time for the user, who does not have to run around chasing
> links, carefully checking for a kernel match, downloading, patching,
> building and installing a single purpose kernel, and bringing it up on
> a machine that would probably have only required one click on the new
> filesystem option otherwise.  The considerable time thus saved can be
> invested profitably in running test cases and filing bug reports.

Bug reports against a 3-6 months old snapshot of a filesystem being 
under heavy development.

Ted said back in August in the announcement of an ext4 patchset:

"As before I've also released updated the patch set vs. the 2.6.26 stock
 kernel, for those people who don't want to play with development 
 kernels but who still want to test out ext4." [1]

When running stable kernels you still have to patch, build and install
a single purpose kernel for testing ext4 although ext4 is in mainline.

>...
> > This is not meant as a statement on the quality of ext4 or btrfs, or any 
> > comparison of the development times of ext4 and btrfs, but for ext4 the 
> > advantages Andrew thinks would happen with an early btrfs merge do not 
> > seem to have happened.
> 
> Are you sure about that?  I see 33 messages on linux-ext4 yesterday,
> from a broad range of contributors.  Versus eight from a much narrower
> range of contributors, Oct 4 a year ago.

There are lies, damn lies, and statistics.  ;)

Single day statistics about mailing list postings are not very good
indicators for anything.

And since linux-ext4 is for all of ext2/ext3/ext4 the data you gave 
could equally be used to prove that ext3 recently became much more
buggy or that ext2 development vastly increased...

> There is little question that an early merge helps both developers and
> users employ their time more efficiently,

Regarding users see my comment above.

Regarding developers it would be interesting to hear some experiences 
from ext4 developers about their experiences (or get a pointer to them 
in case I missed that they already expressed it somewhere).

> once a project is past the
> point where we wonder about its value and/or viability.  In my opinion,
> Btrfs clearly has both.
>...

2 years ago ext4 was in a similar situation of being regarded as an 
important future filesystem.

> Regards,
> 
> Daniel

cu
Adrian

BTW: My comments are not in any way meant against btrfs or ext4.
     I just question the advantages of merging them early.


[1] http://lwn.net/Articles/294784/

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [RFC] Btrfs mainline plans

[Theodore Tso]

I've refrained from commenting on this thread mainly because I've been
hugely busy between (a) Linux Foundation 2009 budget planning, (b)
preparing for the Linux Foundation end user summit, (c) making sure
ext4 tree was ready for the opening of the 2.6.28 merge window, and a
million other things --- and because I think this thread is largely
pointless.  At the end of the day, it's mostly Adrian arguing against
early merging, and if Andrew is favor of merging btrfs this point, and
given Linus preferences for early merging of things like device
drivers, it's going to happen regardless of Adrian's opinions.

>From the point of the filesystem, it's all upside to be merged into
the kernel mainline.  We've *always* said that we strongly discourage
out-of-tree kernel modules, whether it's device drivers or externally
maintained filesystems whether it's binary VxFS or ClearCase
filesystem (which is GPL'ed and yet maintained out of tree for a
variety of reasons).  One of the things that we do in order to
strongly discourage out of tree drivers/filesystems is that we
constantly make changes to the API, without regard to making life easy
to the out-of-tree kernel module.  Most of the time the changes are
justified --- although sometimes people have suspected that some
changes made had benefits that were so marginal that it seemed that
the main justification was to screw over externally maintained
drivers/filesystems.  Whether or not that's true, the official party
line is that we show no mercy towards externally maintained device
drivers (even ones that are GPL'ed); the Right Answer is that they
should be part of the mainline kernel.

If that is true, there are very few justifications for keeping a
proposed kernel module out of the tree.  The main consideration is
whether the code will, in the long term, be maintainable.  There is
some minimum level of quality that is needed, although there is some
disagreement about what that level is; but probably what is more
important is the reputation of the maintainer and how trusted that
maintainer is to fix any problems which come up.  If we're going to be
honest with ourselves, that's probably one of the reasons why Reiser4
was never accepted.  Sure, there were technical problems with the
code, but at the main day, the primary problem was that Hans didn't
play well with others, especially those who tried to send him
criticism and/or suggestions about how is code would be improved.  As
a result, it drove away people who were willing to review his code,
and no one was willing to speak up in his defense or give him the
benefit of the doubt.

As far as ext4 is concerned, being in the mainline was all upside, and
I believe that having in the kernel *did* accelerate its progress.  It
meant that kernel-wide API changes were applied automatically, and it
meant that kernel developers who wanted to try out ext4 could do so
quite easily.  Yes, in the past two releases I started maintaining
patchsets against a stable kernel; this was mainly to support those
users who didn't want to follow the latest git releases --- and that
was a reflection that ext4 was mature enough that there were stable
kernel users who were interested in using ext4.  I could have used the
-stable infrastructure, but ext4 was changing so rapidly that it was
easier just to maintain a full patchset.  As a matter of fact,
starting with 2.6.27, given that we'll be renaming ext4dev to ext4 in
the 2.6.28 mergeset, the plan is that we'll be submitting patches to
the -stable series.

Yes, ext4 didn't go as quickly as I would have liked, but part of the
problem was I personally didn't have enough time to review the patches
being created by the various ext4 developers, and I wasn't about to
merge patches until they were ready.  We didn't have enough senior
developers on ext4, and it took a while for some of the developers
assigned to the project to get up to speed.  (I was the most senior
developer, but I've never had time assigned by my employer to work on
ext4; it has always something I did on my own time, often late at
night (hint: check the time this mail was sent, and when the last ext4
patchset was sent out last night).  Fortunately, at this point a
number of developers like Aneesh have become comfortable with the
code, and good at writing patches that don't require major review and
changes, and the addition of engineers hired by Red Hat, such as Eric
and Val, have also helped immensely.

As far as btrfs is concerned, one of the things that you may not know
is that about a year ago (on November 12-13, 2007), a small group key
filesystem developers, that included engineers employed by HP, Oracle,
IBM, Intel, HP, and Red Hat, and whose experience included working
with a large number of filesystems: ext2, ext4, ext4, ocfs2, lustre,
btrfs, advfs, reiserfs, and xfs came together for a two day "next
generation filesystem" (NGFS) workshop.  At the end of the that
workshop, there was unaminous agreement (including from yours truly)
that (a) Linux needed a next generation filesystem to be competitive,
(b) Chris Mason's btrfs (with some changes/enhancements discussed
during the workshop) was the best long-term solution for NGFS, and (c)
because creating a new enterprise filesystem always takes longer than
people expect, and even then, it takes a while for enterprise users to
trust a new filesystem for their most critical data, ext4 in the next
generation of filesystems was needed as the bridge to the NGFS.

The reason why we made these recommendations was not to influence open
source developers (which is why we haven't really talked about it a
lot in venues like the LKML) but as recommendations to the management
of the above-mentioned for assigning resources to the project.  (One
of the recommendations we made was that a critical success factor was
that knowledge about the filesystem must be spread throughout multiple
vendors and distributions.)  But I think it is fair to say that btrfs
isn't just a private a project of a single Linux kernel developer, but
rather the design has been discussed and reviewed by a large number of
experienced filesystem architects.  What *is* important is that Chris
is a well-known kernel developer who is trusted to create and maintain
quality kernel code, and his employer *has* apparently given him
enough time that he can do a lot of personal, hands-on development.

Given btrfs's current status, in terms of its functionality, even its
format is not fully cast into stone yet, and given Chris's reputation
and skills as a kernel devleoper, my personal opinion is that we would
not be making a "special case exception" for btrfs to get it into
mainline, but rather something which makes completely good sense.

At the end of the day, though, it's not my opinion or Adrian's opinion
that matters --- it's really Linus's call.  But if Linus were to ask
my opinion, I would say, "Yes, absolutely --- we should merge btrfs
into mainline."

Regards,

							- Ted