Linux 2.4 usage statistics

Linux 2.4 usage statistics

[Willy Tarreau]

Hi all,

I would be very interested in getting feedback about kernel 2.4 usage.
Please, please, I do not want this thread to degenerate into a useless
flamewar or "mine is bigger" discussion. For this reason, I would like
that participants reply privately to me so that this mail does not turn
into a thread.

I know 2.4 is still used in a lot of sensible areas, although far less
than 2.6 nowadays. At least I know that several appliance makers/sellers
still use it in new products, and even very old versions sometimes. What
I'd like to estimate is the way it is used, if it is systematically
patched with local add-ons, frequency of updates (if any), etc... so
that I can adapt my process if needed. For instance, I don't release
any -rc for stable releases because I'm convinced that nobody will ever
test them, but I have no problem being proved wrong.

If I get enough replies to build up a statistic synthesis, I'll try to
summarise it (eventhough 73% of statistics are all made up :-) ).

This "poll" is not meant to dictate what will get merged next (we're in
feature freeze anyway), but it helps me know your usage better, to try
to serve you better. Also, if a majority of people report the same
problem or lacking feature for which a trivial fix is know, it can be
studied.

Basically what I'd like to know is the following. You can use the proposed
responses as a guide, but they're not mandatory. Also if possible and adapted,
please report number of concerned units :

1) where do you use it ?
   - PC turned into network equipment (router, LB, BGP route reflector, ...)
   - security equipment (firewall, vpn, ids/ips, traffic analyzer, ...)
   - proxy services (proxy, smtp relay, reverse-proxy, anti-virus/spam, ...)
   - storage/directory server (NFS, LDAP, logging, ...)
   - multi-function server (proxy/fw/nfs/mail/...)
   - monitoring / remote access
   - desktop/workstation
   - laptop (if recent, what model ?)
   - dedicated appliances (that you may be designing/selling)
   - soho embedded systems (eg: wifi routers, ...)
   - PDA ?
   - other ?

2) how critical are the systems it runs on ?
   - medical (people's health depend on them) ?
   - finance (massive amounts of money pass through them) ?
   - business-critical (you may go bankrupt if it fails too often) ?
   - mission-critical (you may loose your job if it fails too often) ?
   - security-critical (security gets degraded when it fails) ?
   - users get angry at you when it fails (admin, or do this for a living) ?
   - not that important (you may reboot when you decide to do so) ?
   - no importance at all (eg: home usage, experimentation, ...) ?

3) how many users are you serving with it approximately, all systems
   included ? Or better, how many people will notice the failure at once
   if any ? Note: do not count web population as users, but use maximum
   concurrent users instead.
   - < 10
   - < 100
   - < 1000
   - < 10000
   - more ?

4) what version are you running ?
   - latest or close to that (eg: 2.4.36.x)
   - latest was available last time you upgraded
   - somewhat old mainline (2.4.32-2.4.34)
   - quite old mainline (2.4.21..2.4.31)
   - very old mainline (2.4.2, 2.4.17, ...)
   - debian's 2.4.27
   - rhel 3's 2.4.21
   - my other standard distros's 2.4 kernel
   - an embedded distro's heavily patched 2.4 (port to other archs, ...)

5) have you applied any patches ?
   - provided with the distro
   - security: pax/grsec/ow, capabilities, ...
   - performance: epoll, variable_hz, jiffies_64, preempt, O(1), ...
   - network: tcp md5, transparent proxy, netfilter tcp_window_tracking...
   - drivers: very specific to your platform, self-added PCI-IDs, vendors'
   - filesystems: updated jffs2+mtd, squashfs, cifs, fuse, ...
   - security/stability fixes backported from later 2.4 versions
   - other security/stability fixes or improvements
   - drivers backported from 2.6
   - other ?

6) how often do you update (may translate into avg uptime) ?
   - every release
   - every major release after a few dot-versions
   - only when you see an important (to you) security fix
   - same as well as with reliability fixes
   - at least once a year
   - only if you need to re-install
   - after a lot of careful revalidation
   - never

7) how do you test ?
   - pre-production environment in equivalent conditions
   - long-term reliability stress-tests (days to weeks)
   - quick regression testing before production
   - you test in production
   - no test at all, blind upgrade
   - other ?

8) why have you not upgraded to 2.6 yet (including Adrian's 2.6.16.X ?)
   - not needed, all features OK and hardware still 100% compatible
   - you have to change user-space tools
   - you don't feel comfortable with 2.6's patch pace (but 2.6.16 is quite
     comparable to 2.4)
   - you do not trust 2.6 enough yet ? (why ?)
   - missing features you had in 2.4 mainline that are not in 2.6 anymore ?
   - missing features you had in your 2.4 patches that do not exist in 2.6 ?
   - drivers not existing anymore in 2.6 ?
   - drivers not working anymore in 2.6 ?
   - performance regression in 2.6 ?
   - you find 2.6 more complicated ?
   - other ?

9) when do you plan to switch to 2.6 ?
   - it has already begun (and as a supplement, if you had to do so because
     2.4 did not work anymore for you for other reasons than hardware
     support and features, please indicate which ones)
   - very soon (<1 year)
   - not before 1 year
   - next installations
   - next batch of major upgrades
   - when your hardware is not compatible anymore
   - when you can't build 2.4 with your distro's gcc
   - when 2.6 supports feature/driver XXX
   - when 2.6 is more reliable (example ?)
   - when 2.6 development slows down
   - the latest possible (why ???)
   - never (why ??? and what would you switch to then ?)

10) any comments or suggestions about current release process, minor issues
    that you are facing each time such as external patches that do not apply,
    recurrent problems on all of your setups, etc... ?


Well, that's all. You do not need to respond precisely, and once again,
I'm just trying to get the figure.

And please do not forget! Reply to me directly, do not pollute the list,
that way you will refrain the random geeks from comparing their setups
to other ones.

Thanks!
Willy

Re: Linux 2.4 usage statistics

[Adrian Bunk]

Just a note that some "how many" and "which version exactly" data is 
available from [1] which is generated from data users send through a 
cron job with data from nearly 5000 machines reported during the last 
60 days.

When clicking on a version (e.g. "2.4.27") you can also see which of 
them are distribution kernels.

Kinda shocking that 14% of these machines are not running kernel 2.6 ...

Considering how this data is generated it obviously shows only part of 
the picture, and your survey will hopefully bring data what we could do 
for making kernel 2.6 more attractive (not meant against your work, but 
we should aim at bringing users to 2.6).

cu
Adrian

[1] http://counter.li.org/reports/systemstats.php

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.4 usage statistics

[Willy Tarreau]

Hi Adrian,

On Mon, Apr 28, 2008 at 11:56:31PM +0300, Adrian Bunk wrote:
> Just a note that some "how many" and "which version exactly" data is 
> available from [1] which is generated from data users send through a 
> cron job with data from nearly 5000 machines reported during the last 
> 60 days.

I've already seen this one sometime ago, but realized that you will never
get stats from sensible systems there. For instance, none of my customers
running linux in production would ever accept to permit an HTTP communication
between any of their servers with another one on the need, and especially
when it comes to reporting stats about *their* versions.

Also, I recently realized that several high-grade commercial products still
ship with 2.4 in it. I even know about one which never said it used Linux,
running on MVL 2.4.2 :-)

Of course those ones do not care a dime about recent versions. But it looks
like the falling curve has reached a stabilization point, because such
products would have had hundreds of opportunities to upgrade. Reason why
I'm wondering and asking them directly.

> When clicking on a version (e.g. "2.4.27") you can also see which of 
> them are distribution kernels.
> 
> Kinda shocking that 14% of these machines are not running kernel 2.6 ...
> 
> Considering how this data is generated it obviously shows only part of 
> the picture, and your survey will hopefully bring data what we could do 
> for making kernel 2.6 more attractive (not meant against your work, but 
> we should aim at bringing users to 2.6).

100% agreed, and it's the orientation of the survey. At least 2.6.16 could
be a first step for those who need high code stability.

Speaking for my case, at Exosec we still use 2.4 a lot. Main reason is that
we are used to apply a lot of patches. And maintaining a kernel which does
nearly not change in 6 months is really a joy. I have already thought about
moving to 2.6.16, but I would have had to port my patches, and was not
satisfied by the crapp^Wold scheduler which caused real performance issues
for my workload. Since I would have gained nothing in this operation, it
was easier to stick to 2.4.

I'm waiting for other people's excuses now :-)

I'm really tempted by making a new attempt with 2.6.25, but let's let it
settle down first.

> cu
> Adrian
> 
> [1] http://counter.li.org/reports/systemstats.php

Cheers,
Willy

Re: Linux 2.4 usage statistics

[Stefan Richter]

Willy Tarreau wrote:
> Speaking for my case, at Exosec we still use 2.4 a lot. Main reason is that
> we are used to apply a lot of patches. [...]  I have already thought about
> moving to 2.6.16, but I would have had to port my patches, [...]
> Since I would have gained nothing in this operation, it
> was easier to stick to 2.4.
> 
> I'm waiting for other people's excuses now :-)

A variation of the theme, which I as maintainer of some drivers in 2.6 
heard by way of occasional questions on subsystem mailinglists or 
off-list:  Kernel customizations for embedded systems in combination 
with a custom userland.  Moving all this software over to 2.6 was not 
considered worth the effort.

Another reason which I read here and there and don't know how real the 
issue is:  2.6's minimum possible footprint is said to be unable to 
compete with 2.4 for really resource restricted applications.

Could it actually be that 2.4 outnumbers 2.6 in embedded installations?

(I don't work with 2.4 myself and I don't know the domain of embedded 
systems.)
-- 
Stefan Richter
-=====-==--- -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.4 usage statistics

[Lee Revell]

On Tue, Apr 29, 2008 at 3:06 PM, Stefan Richter
<stefanr@s5r6.in-berlin.de> wrote:
>  Another reason which I read here and there and don't know how real the
> issue is:  2.6's minimum possible footprint is said to be unable to compete
> with 2.4 for really resource restricted applications.
>
>  Could it actually be that 2.4 outnumbers 2.6 in embedded installations?

Possibly.  However many embedded applications have real time
constraints so are using 2.6 these days.  Both vanilla and -rt patched
2.6 beat the pants off 2.4 in this area.

Lee

Re: Linux 2.4 usage statistics

[Jean Delvare]

Hi Willy,

On Mon, 28 Apr 2008 23:18:12 +0200, Willy Tarreau wrote:
> I'm really tempted by making a new attempt with 2.6.25, but let's let it
> settle down first.

You might want to wait a bit more and pick a version which will end up
in either SLES or RHEL. These tend to be maintained for longer than the
other versions. For example SLES10 is based on kernel 2.6.16, which is
still maintained by Adrian (or at least was maintained for a long time
- I don't know what the current status is.)

-- 
Jean Delvare

Linux 2.4 usage statistics (results)

[Willy Tarreau]

Hi all,

I've tried to summarize the usage reports I got for kernel 2.4.

[ Note: some of the responders asked to be notified when I send the
  results, so I've Bcc'd the responders so that they can get the
  results without being polluted by potential responses.            ]

About 22 useful responses. It is not possible to report accurate numbers
because most participants did not themselves have accurate numbers. I
would say that based on the responses (and a few cases I know), 2.4 usage
approximately follows this distribution :

 - old recycled laptops at home, or PDA/thin clients - ~5%
   They rely on whatever kernel managed to boot on them, then they never
   got updated anymore. Security generally not too much of an issue, no
   plan to update to newer 2.4, let alone 2.6. The hardware will die first.
   Sometimes a few patches were added to support one particular component
   or enable one feature specific to the use case. These boxes have the
   lowest level of criticity.

 - desktop PCs, monitoring stations - about the same as previous ones, with
   an increased level of criticity. Generally users have not upgraded simply
   because "it works". Those are systems with known fixed hardware and usage
   pattern. Most often they serve as X11 terminals and/or browsers, and do
   not require many updates. They cause trouble to the user(s) when they
   die and no benefit is expected from an upgrade to 2.6. Future installation
   (same or replaced hardware) will generally be on 2.6.

 - general purpose servers : regularly updated - about 50%.
   The pattern is most always the same. A lot of services are installed,
   among which WWW, Mail, DNS, Samba, NFS. The server never experiences
   any outage, and a moderate amount of people would be affected if a
   problem happened. Almost always up to date with latest 2.4. Reason
   for not upgrading to 2.6 generally is lack of need and time, as well
   as risks of regressions which would get a lot of users angry. Users
   per server are in the range of 10 to 1000. Generally some migration
   tests are in progress. Early attempts at 2.6 failed due to stability
   problems (GRE tunnels freezing, sparc images limited in size).

 - application specific servers : about 20%. Often mission-critical.
   Generally deployed with latest 2.4, and almost never updated.
   Long stress-tests before production. Generally the number of users
   does not mean anything, it's the process the system is involved in
   which makes sense. Pre-press workflow applications for daily
   newspapers and weather broadcasting for TV channels were reported.
   Reliability is the #1 requirement, and generally recent tests in
   2.6 have not been much satisfying. One reported DRBD 0.6 very reliable
   in 2.4, while DRBD 0.7 not as much in 2.6. Another one reported that
   the application needed to be ported to correctly work in 2.6.
   Interestingly, in all cases, the high level of criticity implies
   that 2.6 is still being evaluated, in the event that one day there
   is no choice (eg: due to hardware compatibility).

 - routers/firewalls/VPN/IDS : about 10%. Longest uptimes up to 5-600 days
   in business environments, implying kernel not often updated (avg once a
   year). For personal use, it's where the updates are the most common
   (probably due to the quick reboot and low impact of a short outage).
   One user reported a number of these boxes deployed in finance/business/train
   environments with high level of criticity, receiving occasional updates
   during dead hours. Generally no upgrade to 2.6 is planned (though I think
   it is where it might be the least painful). One user reported missing
   devfs in 2.6, too big kernel size, and no clear upgrade path. We might
   help them switch by enumerating the small number of changes in such a
   networked environment. What may be the most obscure in business environment
   is the lack of sight of long term reliability. I personally know about
   several firewalls I have deployed 5 years ago in a finance environment
   which see 1 million unique clients a day. It has been discussed about
   migrating them to 2.6 but it seems like the only way to know how long
   they will survive is to test in production, which is generally not
   acceptable. So leaving them on 2.4 is often the solution.

 - embedded systems : about 10% of the reports, probably more in terms of
   number of systems. They often expect very long uptimes (two of them
   reported rebooting less than once a year). This is either dictated by
   the fact that consumer electronics get bad reputation when failures or
   updates happen too often, or because the big customer does not want
   to update something which works. Reported usages include outdoor
   information displays for trains and busses, various consumer electronic
   devices including TV sets, monitoring systems and building security
   devices (eg door openers). Upgrade to 2.6 not needed nor planned at
   all for some, and planned for others to gain better hardware support.
   Some are still testing but experiencing regressions (eg: parport). In
   all cases, the build process has to be massively redesigned and this
   costs a lot of time and money. Not surprizing that some major network
   equipement manufacturer still ships device with and old 2.4.2 in them.

It is worth noting that several users still using 2.4 roll their own distro,
which will need to be substantially updated to support 2.6. This seems to be
one showstopper too, especially when the distro was designed for easy remote
and centralized upgrades. In this case, what seems to happen is that the old
version continues to live at existing locations, and a new version appears
with support for 2.6 for newer systems. But from an organisational point of
view, it's not always easy to manage two branches of a distro when only one
has been enough for years.

Surprizingly, it appears that very few users patch their kernels. Those who
do so have their own distros or build kernels for large amounts of machines
(eg: consultants doing so for their customers). Many of the reported patches
have found their way in mainline 2.6, but some have definitely vanished (eg:
devfs,linux-abi, kstat, umsdos, lvm). Among the patches, GRsecurity and PaX
were reported several times. Some special-purpose patches are included in
appliance products.

Vendor drivers are the only ones installed by most users who patch (eg:
3Ware 9xxx, sk98lin, bnx2). One user reported an annoying problem with USB
for which I realized I've been having a patch in my own tree for years, to
the point I completely forgot the problem existed. It is related to
USB-storage, where unplugging then replugging a device assigns it a new
SCSI drive name. I may merge it into 2.4.37.

Based on that and on the workflow people took the time to explain, I
realize that the distinction between -pre and -rc is useless (ding! Linus
if you read this, don't beat me). In fact, either people want absolute
reliability and they pick one kernel from the stable 2.4.X.Y branch, or
they want more recent updates and they simply do their shopping in the
-master branch, which is fairly easy thanks to the Gitweb interface.
But there are almost no testers in 2.4, just users. That means that I
don't have to expect immediate feedback when posting a pre-release. And
it has happened several times that I got a build error report several
weeks after the release.

Also, since most people do not update more than 1/2 times a year, it's
not very useful to have more than 1/2 new versions a year, expecially
since we have the stable release. For this reason, I think I will issue
stable releases a bit more often for users to quickly get their fixes,
but progressively increase the delay between major releases. Those ones
will only be issued with new PCI IDs, major driver updates, compiler
support, etc...

Last, I think users would benefit from an inventory of functional changes
between 2.4 and recent 2.6, with a reversible upgrade path, so that they
can experiment with 2.6 on their production machine and immediately go
back to 2.4 in case of trouble.

Well, I hope it was not too much boring!

Regards,
Willy