Help to know the stable ver of ext4 for commercial app.

Help to know the stable ver of ext4 for commercial app.

[Wuqixuan]

Hi all, 

   Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts: 
 
    1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not. 
    2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ? 
 
    Any answer is helpful and welcome. 
    Thanks a lot. 
 
Best Regards!
Wuqixuan

Re: Help to know the stable ver of ext4 for commercial app.

[Theodore Ts'o]

On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
> 
>    Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts: 
>  
>     1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not. 
>     2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ? 

Well, it's _possible_ to use 2.6.34 as a base for a production server.
Red Hat's RHEL 5 was based on kernel of approximately that vintage
(with many, many patches).  Google announced it was upgrading its data
centers to use ext4 just about the time I was hired at Google[1], in
January 2010, and 2.6.34 dates from May 2010.

[1] http://lists.openwall.net/linux-ext4/2010/01/04/8

The team at Taobao was using a similar vintage kernel if I recall
correctly, at least at one point in time.  I'm not sure if they have
since upgraded to a newer kernel.

However, at Google, we used a huge number of patches to fix and
stablize ext4.  The patches were pushed upstream over a 18 months or
so, but not all of them were backported into 2.6.34.x kernels.  I
believe you will find the same story at Red Hat and at Tao Bao.


Part of that is because the stable kernel maintainers will generally
just drop a patch if it doesn't apply cleanly.  In order to determine
which patches are applicable, and to adopt and backport patches, you
really will need one or more ext4 developers on staff, such as Eric
Sandeen and his colleagues for Red Hat, or me and my colleages for
Google, or Tao Ma and his team at Taobao.

Whether or not it's good enough for your streaming media application
is a different question.  That's a very restricted use case, and it
might be good enough.   You should test it and see for yourself.  

Regards,

						- Ted

Re: Help to know the stable ver of ext4 for commercial app.

[Wuqixuan]

Hi Ted, 

 Thanks a lot for your information. Has some doubt below. 

> On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
>>
>>    Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts:
>>
>>     1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not.
>>     2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ?

>  Well, it's _possible_ to use 2.6.34 as a base for a production server.
>  Red Hat's RHEL 5 was based on kernel of approximately that vintage
>  (with many, many patches).  Google announced it was upgrading its data
>  centers to use ext4 just about the time I was hired at Google[1], in
>  January 2010, and 2.6.34 dates from May 2010.

>  [1] http://lists.openwall.net/linux-ext4/2010/01/04/8

>  The team at Taobao was using a similar vintage kernel if I recall
>  correctly, at least at one point in time.  I'm not sure if they have
>  since upgraded to a newer kernel.

>  However, at Google, we used a huge number of patches to fix and
>  stablize ext4.  The patches were pushed upstream over a 18 months or
>  so, but not all of them were backported into 2.6.34.x kernels.  I
>  believe you will find the same story at Red Hat and at Tao Bao.

>  Part of that is because the stable kernel maintainers will generally
>  just drop a patch if it doesn't apply cleanly.  In order to determine
>  which patches are applicable, and to adopt and backport patches, you
>  really will need one or more ext4 developers on staff, such as Eric
>  Sandeen and his colleagues for Red Hat, or me and my colleages for
>  Google, or Tao Ma and his team at Taobao.

   From these two paragraphs, my understanding is, during the 18 months 
about the huge number of patches, all the patches which impact 2.6.34.x 
had been backported into old stable version like 2.6.34.x. If not merge, 
just because those patches is not applicable to or impat that version. I am 
not sure whether I am wrong. 

   Another doubt is that currently if a new patch is merged in upstream, who is to be obligated to 
do backport. The maintainter of ext4 (you) or the stable branch maintiner (like 
maintiner of 2.6.34.x, i think he is from windriver) ? I saw even in 2013-01-16, 
still one checkin (commit: 25772970966c268c16ec5c0f9d02449f401663c1) 
in 2.6.34.14. Does it mean that if one patch is merged in upstream, somebody 
generally will be obligated to do backport to old stable branches within some period.

   So for a conclusion, can I think, no valuable patch( or not many patch) is missing to be 
backport from upstream high version currently ? 

>  Whether or not it's good enough for your streaming media application
>  is a different question.  That's a very restricted use case, and it
>  might be good enough.   You should test it and see for yourself.

Regards,

                                                - Ted

Re: Help to know the stable ver of ext4 for commercial app.

[Andreas Dilger]

On 2013-02-02, at 0:20, Wuqixuan <wuqixuan@huawei.com> wrote:
> 
> Thanks a lot for your information. Has some doubt below. 
> 
>> On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
>>> 
>>>   Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts:
>>> 
>>>    1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not.
>>>    2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ?
> 
>> Well, it's _possible_ to use 2.6.34 as a base for a production server.
>> Red Hat's RHEL 5 was based on kernel of approximately that vintage
>> (with many, many patches).  Google announced it was upgrading its data
>> centers to use ext4 just about the time I was hired at Google[1], in
>> January 2010, and 2.6.34 dates from May 2010.

If you are looking at stability for a commercial application, I think you would be best off to choose something like RHEL6 2.6.32 which has back ported all of these patches for ext4 already and will continue to get fixes for years to come. 

Next best would be to pick something like FC18 3.8(?), which also has the needed patches, but will not be supported for as long a time.  This is definitely still better than picking the vanilla kernel of the same version. 

That way, you can focus on your application, and not on becoming a kernel maintainer...

Cheers, Andreas

Re: 答复: Help to know the stable ver of ext4 for commercial app.

[Theodore Ts'o]

On Sun, Feb 03, 2013 at 07:13:19AM +0000, Wuqixuan wrote:
> 
>      Thanks, I will check and compare the latest RHEL6 2.6.32 ext4 and the 2.6.32.x and 2.6.34.x in the vanilla kernel. 

I think you will find that they are quite different.

>      BTW, even now I can see in 2.6.32.x version in vanilla kernel,
>still there are some patches are checked in. But not very
>frequently. There are 5 in 2011, and 4 in 2012.

In general, these are the patches which applied trivially, so it woud
not require any special expertise to backport them.  In many cases,
these are what could be called "one-liners".

>So do you mean Redhat
>does not checkin all the patches which backport in RHEL6 to the
>vanilla kernel ?

That's correct.  It is not their responsibility to update generic
2.6.32.x kernels.  Keep in mind that when distributions talk about
"upstream first", what they mean is that new features and new drivers
get developed in the tip of the latest development upstream kernel,
and then get backported to whatever ancient "enterprise kernel" (i.e.,
2.6.32.x, et. al) that their customers pay $$$ to for them to support.

In some cases, when distributions fix a bug in their ancient 2.6.32.x
kernels for one of their paying customers, they will make point of
checking to see if the bug still applies in the development upstream
kernel, and if it does, they will submit a patch to the upstream
kernel.  Those are the distributions I will tend to help out.  But
when a company like Montavista comes out of nowhere, without having
contributed anything to ext4, and then asks for free engineering help
for their commercial kernel, it's likely that they free engineering
help they will receive will be minimal....

>      And do you know how about the stability of other distrubutions, like Windriver 4 sp3 ? 

You should really ask Windriver, and not the upstream lists.  Keep in
mind that for an upstream development engineer, commercial kernels
tend to be considered evolutionary dead ends....

							- Ted

Re: Help to know the stable ver of ext4 for commercial app.

[qixuan wu]

On Sun, Feb 3, 2013 at 8:43 PM, Wuqixuan <wuqixuan@huawei.com> wrote:

> On 2013-02-02, at 0:20, Wuqixuan <wuqixuan@huawei.com> wrote:
>>
>> Thanks a lot for your information. Has some doubt below.
>>
>>> On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
>>>>
>>>>   Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts:
>>>>
>>>>    1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not.
>>>>    2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ?
>>
>>> Well, it's _possible_ to use 2.6.34 as a base for a production server.
>>> Red Hat's RHEL 5 was based on kernel of approximately that vintage
>>> (with many, many patches).  Google announced it was upgrading its data
>>> centers to use ext4 just about the time I was hired at Google[1], in
>>> January 2010, and 2.6.34 dates from May 2010.
>
> If you are looking at stability for a commercial application, I think you would be best off to choose something like RHEL6 2.6.32 which has back ported all of these patches for ext4 already and will continue to get fixes for years to come.
>
> Next best would be to pick something like FC18 3.8(?), which also has the needed patches, but will not be supported for as long a time.  This is definitely still better than picking the vanilla kernel of the same version.
>
> That way, you can focus on your application, and not on becoming a kernel maintainer...

Thanks andreas, will check the RHEL6 2.6.32 ext4 and compare with
Windriver 4.3 (2.6.34.10) which we are ready to use. Do you have any
idea of ext4 of Windriver 4.3 (2.6.34.10) ?

Best Regards.
Wood.

Re: Help to know the stable ver of ext4 for commercial app.

[Li Zefan]

On 2013/2/5 1:52, qixuan wu wrote:
> On Sun, Feb 3, 2013 at 8:43 PM, Wuqixuan <wuqixuan@huawei.com> wrote:
> 
>> On 2013-02-02, at 0:20, Wuqixuan <wuqixuan@huawei.com> wrote:
>>>
>>> Thanks a lot for your information. Has some doubt below.
>>>
>>>> On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
>>>>>
>>>>>   Currently we are choocing ssd/sata file system using for stream media application. We are using 2.6.34.10 (can be upgraded to 2.6.34.14). We are consider of ext3 or ext4.  Even in wikipedia, it mentions ext4 is stable from 2.6.28, I still have some doubts:
>>>>>
>>>>>    1. Whether 2.6.34.10 of ext4 is stable enough for commercial use or not.
>>>>>    2. Are there many mature commercial application on ext4 on 2.6.34.10 or before? Can tell some famous company or app name ?
>>>
>>>> Well, it's _possible_ to use 2.6.34 as a base for a production server.
>>>> Red Hat's RHEL 5 was based on kernel of approximately that vintage
>>>> (with many, many patches).  Google announced it was upgrading its data
>>>> centers to use ext4 just about the time I was hired at Google[1], in
>>>> January 2010, and 2.6.34 dates from May 2010.
>>
>> If you are looking at stability for a commercial application, I think you would be best off to choose something like RHEL6 2.6.32 which has back ported all of these patches for ext4 already and will continue to get fixes for years to come.
>>
>> Next best would be to pick something like FC18 3.8(?), which also has the needed patches, but will not be supported for as long a time.  This is definitely still better than picking the vanilla kernel of the same version.
>>
>> That way, you can focus on your application, and not on becoming a kernel maintainer...
> 
> Thanks andreas, will check the RHEL6 2.6.32 ext4 and compare with
> Windriver 4.3 (2.6.34.10) which we are ready to use. Do you have any
> idea of ext4 of Windriver 4.3 (2.6.34.10) ?
> 

I think you should stop asking about Windriver as you were already told
so twice.

Re: Help to know the stable ver of ext4 for commercial app.

[Eric Sandeen]

On 2/4/13 11:52 AM, qixuan wu wrote:
> On Sun, Feb 3, 2013 at 8:43 PM, Wuqixuan <wuqixuan@huawei.com>
> wrote:
> 
>> On 2013-02-02, at 0:20, Wuqixuan <wuqixuan@huawei.com> wrote:
>>> 
>>> Thanks a lot for your information. Has some doubt below.
>>> 
>>>> On Sat, Feb 02, 2013 at 02:30:11AM +0000, Wuqixuan wrote:
>>>>> 
>>>>> Currently we are choocing ssd/sata file system using for
>>>>> stream media application. We are using 2.6.34.10 (can be
>>>>> upgraded to 2.6.34.14). We are consider of ext3 or ext4.
>>>>> Even in wikipedia, it mentions ext4 is stable from 2.6.28, I
>>>>> still have some doubts:
>>>>> 
>>>>> 1. Whether 2.6.34.10 of ext4 is stable enough for commercial
>>>>> use or not. 2. Are there many mature commercial application
>>>>> on ext4 on 2.6.34.10 or before? Can tell some famous company
>>>>> or app name ?
>>> 
>>>> Well, it's _possible_ to use 2.6.34 as a base for a production
>>>> server. Red Hat's RHEL 5 was based on kernel of approximately
>>>> that vintage (with many, many patches).  Google announced it
>>>> was upgrading its data centers to use ext4 just about the time
>>>> I was hired at Google[1], in January 2010, and 2.6.34 dates
>>>> from May 2010.
>> 
>> If you are looking at stability for a commercial application, I
>> think you would be best off to choose something like RHEL6 2.6.32
>> which has back ported all of these patches for ext4 already and
>> will continue to get fixes for years to come.
>> 
>> Next best would be to pick something like FC18 3.8(?), which also
>> has the needed patches, but will not be supported for as long a
>> time.  This is definitely still better than picking the vanilla
>> kernel of the same version.
>> 
>> That way, you can focus on your application, and not on becoming a
>> kernel maintainer...
> 
> Thanks andreas, will check the RHEL6 2.6.32 ext4 and compare with 
> Windriver 4.3 (2.6.34.10) which we are ready to use. Do you have any 
> idea of ext4 of Windriver 4.3 (2.6.34.10) ?

This really boils down to:

You need to test your application on your hardware.  If you have doubts,
you need a distribution which will provide good support, or be prepared
to maintain it yourself.

Nobody on this list is going to be able to tell you unequivocally which
old kernel on some random distribution is going to work best for your
usecase.

You'll need to be diligent, and invest testing time in this yourself, I think.

-Eric

Re: Help to know the stable ver of ext4 for commercial app.

[Theodore Ts'o]

On Sat, Feb 02, 2013 at 07:20:32AM +0000, Wuqixuan wrote:
> 
>    From these two paragraphs, my understanding is, during the 18 months 
> about the huge number of patches, all the patches which impact 2.6.34.x 
> had been backported into old stable version like 2.6.34.x. If not merge, 
> just because those patches is not applicable to or impat that version. I am 
> not sure whether I am wrong. 

That's not true.  All of the patches which were trivial to apply will
generally be backported into an old stable version --- but that's
ultimately up to the person who stood up and took responsibility for
the doing the stable kernel release for that particular tree.  Greg
K-H did it for a certain set of kernel releases as "long term stable"
releases: 2.6.16, 2.6.27, 2.6.32, 3.4.  He originally did that because
he needed to maintain stable trees for SuSE, and he decided to make
them available as a public service.

Since then, some other people have decided to do the same for other
kernel releases.  Paul Gortmaker from Windriver chose to do this for
2.6.34, probably because he needed it for his employer, and Ben
Hutchings chose to do this for 3.2, because that's the version used by
Debian's upcoming stable kernel version.

>    Another doubt is that currently if a new patch is merged in
> upstream, who is to be obligated to do backport. The maintainter of
> ext4 (you) or the stable branch maintiner (like maintiner of
> 2.6.34.x, i think he is from windriver) ? I saw even in 2013-01-16,
> still one checkin (commit: 25772970966c268c16ec5c0f9d02449f401663c1)
> in 2.6.34.14. Does it mean that if one patch is merged in upstream,
> somebody generally will be obligated to do backport to old stable
> branches within some period.

No one is obligated to do backport any patch; this is all a volunteer
effort.  In some cases, a maintainer may do more work because his
employer very much needs to support a particular file system, but then
he's doing it because it's in the interest of his employer, and he or
she is obligated to do so as long as it continues to be in the
interest of their employer, and as long as the employer is willing to
let those results get published in a form which is useful to the
general public.

Note not all employers have choosen to make their results public.  In
the case of Red Hat, for example, because Oracle was taking their
kernels and using it as a direct competition, and apparently Red Hat
management thought Oracle wasn't contributing sufficiently to the
common effort compared to the economic value they were extracting from
the community, Red Hat management has made the decision to not release
the broken out patches for RHEL kernels.  They do release a
multi-megabyte combined patch, as required by GPL, but that's much
less useful for a competitor to try to use against them.

Part of that is because it takes a huge amount effort to backport
modern patches to really ancient kernels such as 2.6.32 or 2.6.34.
And it's definitely not a fun thing to do, and after you do it, you
have to do a lot of testing to make sure you haven't broken anything.
I had to do this for Google's internal kernel, and it sure wasn't fun,
I can tell you that.  It represents real value, and if you think a
competitor might use it without contributing back to the commons, I
can see why a company might choose not to contribute it back upstream.

>    So for a conclusion, can I think, no valuable patch( or not many
> patch) is missing to be backport from upstream high version
> currently ?

On the contrary, it's very likely that there are missing patches that
were never backported to 2.6.34.  As I said above, backporting patches
that don't just apply because they are complex is a very tough,
miserable, error-prone job, and requires that you very much understand
the code base in question, and that you do a huge amount of testing
afterwards.  For example, there were some key bug fixes that required
quota changes that weren't backported into 2.6.32 or 2.6.34 that are
almost certainly missing from the public stable kernels, because they
require the quota patches as a prerequisite.  Otherwise, you would
have to partially rewrite some of the commits.

Other changes may require changes in the VFS or direct I/O or mm
layers, and dragging in those changes would involve dragging in still
more changes, until it would be easier and safer just to use a newer
kernel.  That's fundamentally the problem with staying back on an
ancient kernel version such as 2.6.34.  People do it because of
"safety", but when you try backporting lots and lots of fixes, the
fact that you have to pull in backports means that you start
decreasing the safety, and you may end up making it worse because no
one is testing these bastardized backports.  The other reason why
people stay back on ancient kernel versions is to save work, but then
doing all of these backports is incredibly painful, and takes a lot of
work, and again, past a certain point, you're better off just jumping
to a newer kernel.

That being said, some of the changes might not matter for your
particular application.  If you aren't using async I/O, or direct I/O,
then patches to fix bugs in AIO or DIO won't matter to you.  Or if are
using fallocate because you can predict in advance how much disk space
will be needed, then some of the bugs in the delayed allocation write
paths might not matter for your particular application.

I hope this helps,

					- Ted

P.S.  The same is true of security patches.  Whether or not all stable
patches get backported to a stable kernel release such as 2.6.32 or
2.6.34 is primarily a question of whether someone is willing to do the
work, and willing to submit it to the stable maintainer.  Which is
another reason why you might want to consider going to a newer
long-term stable kernel, such as 3.2 or 3.4, or paying some company
such as Wind River or Montavista to provide support for not just a
particular kernel, but specific kernel features, and make it _their_
problem to be obliged to make sure all of the needed commits are
backported.

Montavista is currently on my sh*t list, BTW, because one of their
engineers somehow thought I was responsible for providing free
consultation work on a patch which was over 2-3 years old, and
wouldn't even tell me what was wrong, but just "it doesn't seem to
work that well".  Given that Montavista is getting paid by their
customers for the engineering work, asking a volunteer to provide free
engineering work was in very bad taste, and it was even worst taste
not to even give me a well formed bug report, but instead he assumed I
would do code archeology --- for free.  When a distribuion which
doesn't provide any contributions to the upstream mainline kernel,
this is a really good way to piss off a subsystem maintainer....

Re: Help to know the stable ver of ext4 for commercial app.

[qixuan wu]

On Sun, Feb 3, 2013 at 8:43 PM, Wuqixuan <wuqixuan@huawei.com> wrote:
>
>
> ________________________________________
> 发件人: Theodore Ts'o [tytso@mit.edu]
> 发送时间: 2013年2月3日 6:43
> 到: Wuqixuan
> Cc: adilger@sun.com; tm@tao.ma; xiaoqiangnk@gmail.com; Lizefan; linux-ext4@vger.kernel.org
> 主题: Re: Help to know the stable ver of ext4 for commercial app.
>
> On Sat, Feb 02, 2013 at 07:20:32AM +0000, Wuqixuan wrote:
>>
>>    From these two paragraphs, my understanding is, during the 18 months
>> about the huge number of patches, all the patches which impact 2.6.34.x
>> had been backported into old stable version like 2.6.34.x. If not merge,
>> just because those patches is not applicable to or impat that version. I am
>> not sure whether I am wrong.
>
> That's not true.  All of the patches which were trivial to apply will
> generally be backported into an old stable version --- but that's
> ultimately up to the person who stood up and took responsibility for
> the doing the stable kernel release for that particular tree.  Greg
> K-H did it for a certain set of kernel releases as "long term stable"
> releases: 2.6.16, 2.6.27, 2.6.32, 3.4.  He originally did that because
> he needed to maintain stable trees for SuSE, and he decided to make
> them available as a public service.

> Since then, some other people have decided to do the same for other
> kernel releases.  Paul Gortmaker from Windriver chose to do this for
> 2.6.34, probably because he needed it for his employer, and Ben
> Hutchings chose to do this for 3.2, because that's the version used by
> Debian's upcoming stable kernel version.

OK, I got. So generally feature maintainer submit patch to old stable
branch maintainer, and they decide to patch ? Am I correct ?
Because in old stable branch, the whole kernel is very huge. One
person cannot find all the patch in the high version and decide to
patch.

>>    Another doubt is that currently if a new patch is merged in
>> upstream, who is to be obligated to do backport. The maintainter of
>> ext4 (you) or the stable branch maintiner (like maintiner of
>> 2.6.34.x, i think he is from windriver) ? I saw even in 2013-01-16,
>> still one checkin (commit: 25772970966c268c16ec5c0f9d02449f401663c1)
>> in 2.6.34.14. Does it mean that if one patch is merged in upstream,
>> somebody generally will be obligated to do backport to old stable
>> branches within some period.
>
> No one is obligated to do backport any patch; this is all a volunteer
> effort.  In some cases, a maintainer may do more work because his
> employer very much needs to support a particular file system, but then
> he's doing it because it's in the interest of his employer, and he or
> she is obligated to do so as long as it continues to be in the
> interest of their employer, and as long as the employer is willing to
> let those results get published in a form which is useful to the
> general public.
>
> Note not all employers have choosen to make their results public.  In
> the case of Red Hat, for example, because Oracle was taking their
> kernels and using it as a direct competition, and apparently Red Hat
> management thought Oracle wasn't contributing sufficiently to the
> common effort compared to the economic value they were extracting from
> the community, Red Hat management has made the decision to not release
> the broken out patches for RHEL kernels.  They do release a
> multi-megabyte combined patch, as required by GPL, but that's much
> less useful for a competitor to try to use against them.

Actually, i misused the word "obligated",  I just wanted to ask
whether somebody "generally" or "was used to do". I also know the all
people in the community are not the relationship of employment.
Because of less understanding of the machanism of upstream. So please
forgive me.
Now I guess there are some ext4 patches in RHEL latest kernels and not
in the 2.6.32.x branch of vanilla kernel. Also Andreas said in another
mail. I think maybe I need compare the ext4 in RHEL 6 and in the
vanilla kernel. That difference maybe make something clear.

> Part of that is because it takes a huge amount effort to backport
> modern patches to really ancient kernels such as 2.6.32 or 2.6.34.
> And it's definitely not a fun thing to do, and after you do it, you
> have to do a lot of testing to make sure you haven't broken anything.
> I had to do this for Google's internal kernel, and it sure wasn't fun,
> I can tell you that.  It represents real value, and if you think a
> competitor might use it without contributing back to the commons, I
> can see why a company might choose not to contribute it back upstream.

I can imagine. I totally believe that's not funny work. :). Even the
people as strong as you think like that, to say nothing of those like
us.

>>    So for a conclusion, can I think, no valuable patch( or not many
>> patch) is missing to be backport from upstream high version
>> currently ?
>
> On the contrary, it's very likely that there are missing patches that
> were never backported to 2.6.34.  As I said above, backporting patches
> that don't just apply because they are complex is a very tough,
> miserable, error-prone job, and requires that you very much understand
> the code base in question, and that you do a huge amount of testing
> afterwards.  For example, there were some key bug fixes that required
> quota changes that weren't backported into 2.6.32 or 2.6.34 that are
> almost certainly missing from the public stable kernels, because they
> require the quota patches as a prerequisite.  Otherwise, you would
> have to partially rewrite some of the commits.

Ok, seems ext4 of 2.6.34.x in vanilla kernel are missing some
funcationality. Hope not miss too much.

> Other changes may require changes in the VFS or direct I/O or mm
> layers, and dragging in those changes would involve dragging in still
> more changes, until it would be easier and safer just to use a newer
> kernel.  That's fundamentally the problem with staying back on an
> ancient kernel version such as 2.6.34.  People do it because of
> "safety", but when you try backporting lots and lots of fixes, the
> fact that you have to pull in backports means that you start
> decreasing the safety, and you may end up making it worse because no
> one is testing these bastardized backports.  The other reason why
> people stay back on ancient kernel versions is to save work, but then
> doing all of these backports is incredibly painful, and takes a lot of
> work, and again, past a certain point, you're better off just jumping
> to a newer kernel.

Previously I did not realize that the patch backport also related with
VFS/IO/mm layer. I also previously simply thought we can almost
directly copying the 3.x ext4 to reply the folderin 2.6.34.x and
compile without less compiling error. Just because of those incorrect
thought, I simply thought most of patches can be backport easlily.
Hi Ted, how much the proportion or how much of such case is there from
your experience?
But this problem also indicate that the interface of common layer of
kernel also continue to change more. I think that's not good thing.

> That being said, some of the changes might not matter for your
> particular application.  If you aren't using async I/O, or direct I/O,
> then patches to fix bugs in AIO or DIO won't matter to you.  Or if are
> using fallocate because you can predict in advance how much disk space
> will be needed, then some of the bugs in the delayed allocation write
> paths might not matter for your particular application.

I got your pointer. But currently it's hard to say which feature is
sure that will not be used. But for those advanced feature like quota
we can evaluate maybe.

> I hope this helps,

Thank you very much Ted, for your long long information and experience
and your time. So now my understanding is that (Please correct me if I
am wrong.):
1)  When a bugfix is found and patched, mostly apply in the upstream
latest branch.
2)  At the same time, some commercial company are maintaining the
stable long term branch. And continue to backport bugfix in internel
kernel tree, just few are published in the vanilla kernel. Either for
technical reason, or for commercial reason.
3)  So if I directly select 2.6.34.x vanilla kernel, absolutely I can
meet many bugs. And some latest feature also is not there.
4)  But the commercial distribution like RHEL or Windriver should have
many internel backport and more stable.

In another mail, Andreas suggest to directly use RHEL 6, how do you
think about Windriver4 SP3(whose internel kernel is 2.6.34.x), do you
have some information about the stability of it ? Now our product are
selecting WR4.3, previously we are totally no idea whether we can
trust it or not for commercial use directly. Now from your message,
seems commercial distribution is much better than dircectly from
vanilla kernel. So still hope is there :). Want to ask Windriver FAE
directly and compare the code also. If you have info about WR4.3 ext4
about stability, please kindly share to me.  BTW, my "stability"
concept means just one or two defect can be meet and fix in the lab
for normal functionality. Better hard to touch defect in end-customer
in 2-3 year .

I know you also are difficult directly judge without app scenario. OK,
if let you select in WR4.3, what's your intuition or suggestion, ext3
or ext4? :)

Another question, if I directly take from vanilla kernel for
commercial use, at least from which version do you suggestion?

>                                         - Ted
>
> P.S.  The same is true of security patches.  Whether or not all stable
> patches get backported to a stable kernel release such as 2.6.32 or
> 2.6.34 is primarily a question of whether someone is willing to do the
> work, and willing to submit it to the stable maintainer.  Which is
> another reason why you might want to consider going to a newer
> long-term stable kernel, such as 3.2 or 3.4, or paying some company
> such as Wind River or Montavista to provide support for not just a
> particular kernel, but specific kernel features, and make it _their_
> problem to be obliged to make sure all of the needed commits are
> backported.

> Montavista is currently on my sh*t list, BTW, because one of their
> engineers somehow thought I was responsible for providing free
> consultation work on a patch which was over 2-3 years old, and
> wouldn't even tell me what was wrong, but just "it doesn't seem to
> work that well".  Given that Montavista is getting paid by their
> customers for the engineering work, asking a volunteer to provide free
> engineering work was in very bad taste, and it was even worst taste
> not to even give me a well formed bug report, but instead he assumed I
> would do code archeology --- for free.  When a distribuion which
> doesn't provide any contributions to the upstream mainline kernel,
> this is a really good way to piss off a subsystem maintainer....

I totaly agree with you. All the people in the community are just
volunteer. All the maintainer are using the private time to help other
people. Their works should be respected. And ask for any help
absolutely need provide enough information.

When endding the long mail, again say thanks to you, Ted.

Best Regard.
Wood.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help to know the stable ver of ext4 for commercial app.

[Theodore Ts'o]

On Mon, Feb 04, 2013 at 01:39:58AM +0800, qixuan wu wrote:
> 
> OK, I got. So generally feature maintainer submit patch to old stable
> branch maintainer, and they decide to patch ? Am I correct ?

That can happen, but what's much more common is that the subsystem
maintainer will tag a bug fix commits with:

    Cc: stable@vger.kernel.org

in the commit description.  This is a hint that a particular commit
should be grabbed by stable branch maintainers for consideration.

> Previously I did not realize that the patch backport also related with
> VFS/IO/mm layer. I also previously simply thought we can almost
> directly copying the 3.x ext4 to reply the folderin 2.6.34.x and
> compile without less compiling error. Just because of those incorrect
> thought, I simply thought most of patches can be backport easlily.
> Hi Ted, how much the proportion or how much of such case is there from
> your experience?

The vast majority of patches do _not_ backport trivially.  Of the ones
that don't just drop in, roughly a third are due to patch fuzz issues,
i.e., other one-line fixes, or spelling or whitespace fixups.  Another
third or so are relatively simple API changes (i.e., a new parameter
added to a function).  The last third are the ones where pretty major
changes have been made, and the patch may require a partial or
complete rewrite, or a back port of one or more (possibly a dozen) of
prerequisite patches, each of which might be a trivial backport, or
one which requires a moderate amount of work, or might require massive
amounts of work.

And the older the kernel, the harder it gets.  So it's one thing if
you're backporting to a 3.4 kernel, and it will get harder if you are
trying to backport to a 3.0 kernel, and by the time you need to
backport a lot of patches to 2.6.32, the proportion of trivial to
really hard patches starts to shifting more to the really hard.

> But this problem also indicate that the interface of common layer of
> kernel also continue to change more. I think that's not good thing.

Yes, the common layer is constantly changing.  This is considered a
good thing.  For a refutation of why you might think it's not a good
thing, I recommend that you read "stable_api_nonsense.txt" in the
Documentation directory of the Linux kernel sources.

> 4)  But the commercial distribution like RHEL or Windriver should have
> many internel backport and more stable.

They _may_ have internal backports.  It depends on how supported a
particular feature might be.  You need to ask them how much effort
they have put into backporting patches related to a particular
subsystem.  A particular embedded distributions (such as one used by
Montavista and/or Wind Rivier) might not have focused on ext4 if none
of their previous customers have required it.

One thing that you might consider is using a more modern kernel from
the Android git repositories.  The Android team makes a point of using
a fairly recent kernel each time they start a new development cycle;
so for example, Android 4.2 "Jelly Bean" uses a 3.4 kernel, and it
already has ARM chipset and device drivers used in the Nexus 4, 7, and
10.  More importantly, they've been updating it for various bug fixes,
and the Nexus devices use ext4, bug fixes for ext4 (at least which are
noticed by Android users) do get back ported.  (I've worked with the
Android kernel team folks in helping to identify ext4 bug fixes that
needed to be backported into Android kernel git trees from time to
time.)

> I know you also are difficult directly judge without app scenario. OK,
> if let you select in WR4.3, what's your intuition or suggestion, ext3
> or ext4? :)

No idea; you should be asking Wind River.

Either that, or perhaps Huawei should consider establishing a common
kernel team which supports a Linux kernel for all of its products.
(I've heard executives from Huawei talk about their commitment to
Linux at various Linux conferences, so it wouldn't surprise me if
there are kernel folks there already; it's just a matter of getting
them to work together to support all of your various product teams.)

> Another question, if I directly take from vanilla kernel for
> commercial use, at least from which version do you suggestion?

The 3.4 kernel is the most recent long-term supported stable kernel.
Combined with the fact that you can also consult with the kernel git
trees for Android Jelly Bean (version 4.2), it's a pretty compelling
choice.  Of course, it may be that you have a requirement for some
proprietary device driver which is holding you back to an ancient
2.6.34 kernel.  If that's the case, you might want to consider doing
what many experienced product teams have done, which is to simply
demand that your parts manufacturers supply device drivers that are
integated into the upstream kernel --- and if they aren't willing to
do that, that you'll chose some other parts provider.  This is
something IBM and Dell have done for enterprise servers, and it's also
something which manufacuters for consumer electronics products have
started doing.  In the long term, it's a really bad idea to get stuck
back at 2.6.34 vintage kernels.  This is why the Android team make a
point of rebasing to a new upstream kernel at each new development
cycle (i.e., when they went from Honeycomb, to Ice Cream Sandwich, to
Jelly Bean, etc.)

Regards,

						- Ted

Re: Help to know the stable ver of ext4 for commercial app.

[qixuan wu]

On Mon, Feb 4, 2013 at 6:52 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Feb 04, 2013 at 01:39:58AM +0800, qixuan wu wrote:
>>
>> OK, I got. So generally feature maintainer submit patch to old stable
>> branch maintainer, and they decide to patch ? Am I correct ?
>
> That can happen, but what's much more common is that the subsystem
> maintainer will tag a bug fix commits with:
>
>     Cc: stable@vger.kernel.org
>
> in the commit description.  This is a hint that a particular commit
> should be grabbed by stable branch maintainers for consideration.

Totally clear. It's a better way to save subsystem maintainer's time. :)

>> Previously I did not realize that the patch backport also related with
>> VFS/IO/mm layer. I also previously simply thought we can almost
>> directly copying the 3.x ext4 to reply the folderin 2.6.34.x and
>> compile without less compiling error. Just because of those incorrect
>> thought, I simply thought most of patches can be backport easlily.
>> Hi Ted, how much the proportion or how much of such case is there from
>> your experience?
>
> The vast majority of patches do _not_ backport trivially.  Of the ones
> that don't just drop in, roughly a third are due to patch fuzz issues,
> i.e., other one-line fixes, or spelling or whitespace fixups.  Another
> third or so are relatively simple API changes (i.e., a new parameter
> added to a function).  The last third are the ones where pretty major
> changes have been made, and the patch may require a partial or
> complete rewrite, or a back port of one or more (possibly a dozen) of
> prerequisite patches, each of which might be a trivial backport, or
> one which requires a moderate amount of work, or might require massive
> amounts of work.
>
> And the older the kernel, the harder it gets.  So it's one thing if
> you're backporting to a 3.4 kernel, and it will get harder if you are
> trying to backport to a 3.0 kernel, and by the time you need to
> backport a lot of patches to 2.6.32, the proportion of trivial to
> really hard patches starts to shifting more to the really hard.

Seem one third patches backport are difficult. That means the ancient
kernel ext4 is very unstable.

>> But this problem also indicate that the interface of common layer of
>> kernel also continue to change more. I think that's not good thing.
>
> Yes, the common layer is constantly changing.  This is considered a
> good thing.  For a refutation of why you might think it's not a good
> thing, I recommend that you read "stable_api_nonsense.txt" in the
> Documentation directory of the Linux kernel sources.

Implementation of common layer changes constantly is not a problem.
But for interface's constantly change, it stands for design is not
good. If interface change less, then I can directly backport the whole
3.8 ext4 and modify less into my ancient 2.6.34.x kernel. But first I
will study "stable_api_nonsense.txt", maybe I am wrong. :)

>> 4)  But the commercial distribution like RHEL or Windriver should have
>> many internel backport and more stable.
>
> They _may_ have internal backports.  It depends on how supported a
> particular feature might be.  You need to ask them how much effort
> they have put into backporting patches related to a particular
> subsystem.  A particular embedded distributions (such as one used by
> Montavista and/or Wind Rivier) might not have focused on ext4 if none
> of their previous customers have required it.

Yes you are right, maybe for embedded distributions might not have
focused on ext4. If there are huge difference between ext4 of WR4.3
with Android 4.2 "Jelly Bean" kernel. Then maybe we will choose ext3.

> One thing that you might consider is using a more modern kernel from
> the Android git repositories.  The Android team makes a point of using
> a fairly recent kernel each time they start a new development cycle;
> so for example, Android 4.2 "Jelly Bean" uses a 3.4 kernel, and it
> already has ARM chipset and device drivers used in the Nexus 4, 7, and
> 10.  More importantly, they've been updating it for various bug fixes,
> and the Nexus devices use ext4, bug fixes for ext4 (at least which are
> noticed by Android users) do get back ported.  (I've worked with the
> Android kernel team folks in helping to identify ext4 bug fixes that
> needed to be backported into Android kernel git trees from time to
> time.)

Good news. I will compare WR4.3 ext4 with Android 4.2 "Jelly Bean"
kernel in Google's git repositories. But directly use may be
difficult, because of the difference of the ancient kernel and modern
module. So also it indicate that all the patches are not in 3.4
vanilla kernel, otherwise no need backport. Seems 3.4 vanilla kernel
is missing many bugfixes. Am I correct?

>> I know you also are difficult directly judge without app scenario. OK,
>> if let you select in WR4.3, what's your intuition or suggestion, ext3
>> or ext4? :)
>
> No idea; you should be asking Wind River.
>
> Either that, or perhaps Huawei should consider establishing a common
> kernel team which supports a Linux kernel for all of its products.
> (I've heard executives from Huawei talk about their commitment to
> Linux at various Linux conferences, so it wouldn't surprise me if
> there are kernel folks there already; it's just a matter of getting
> them to work together to support all of your various product teams.)

The common kernel absolutely will be based on the higher version newer
than 3.x after 1-2 years. But currently customer is urgently want to
release product to end-customer. If it's not stable, many people will
go the same way of Google/Redhat. That's a mire. But if ext4 is stable
on 2.6.34.x, later we just only need support one filesytem (ext4) in
current release and later common kernel. Otherwise, we need support
ext3(current release) and ext4(common kernel) for duplicating study
effort.

>> Another question, if I directly take from vanilla kernel for
>> commercial use, at least from which version do you suggestion?
>
> The 3.4 kernel is the most recent long-term supported stable kernel.
> Combined with the fact that you can also consult with the kernel git
> trees for Android Jelly Bean (version 4.2), it's a pretty compelling
> choice.  Of course, it may be that you have a requirement for some
> proprietary device driver which is holding you back to an ancient
> 2.6.34 kernel.  If that's the case, you might want to consider doing
> what many experienced product teams have done, which is to simply
> demand that your parts manufacturers supply device drivers that are
> integated into the upstream kernel --- and if they aren't willing to
> do that, that you'll chose some other parts provider.  This is
> something IBM and Dell have done for enterprise servers, and it's also
> something which manufacuters for consumer electronics products have
> started doing.  In the long term, it's a really bad idea to get stuck
> back at 2.6.34 vintage kernels.  This is why the Android team make a
> point of rebasing to a new upstream kernel at each new development
> cycle (i.e., when they went from Honeycomb, to Ice Cream Sandwich, to
> Jelly Bean, etc.)

I think google's strategy is correct, and common kernel will use the
same mode I think later. But before having the common kernel, we still
have to get stuck at one commercial Linux. This is the current policy
from executives, and not because of some perticular device driver.

Best Regards.
Wood.