[STATUS 2.5] May 1, 2002

[STATUS 2.5] May 1, 2002

[Guillaume Boissiere]

This week's status update is available at the usual place:
  http://kernelnewbies.org/status/

A couple items have been merged since last week, including the 
new NTFS driver, the fast walk dcache patch, the start of the 
new framebuffer layer, as well as some more delayed disk block 
allocation bits.

There has also been a lot of work done by various people to 
remove the BKL from many places, which leads to my question:
does anyone has a URL with a list of all the places where the
BKL should eventually be removed and who's working on it?

It seems like it would be most useful if someone was willing
to maintain something like this, but it might be a lot of 
work - I don't know how long the list would be...

Anyway, as usual, if I missed anything, let me know!
Enjoy,

-- Guillaume

--------------------------------------------------
Kernel 2.5 status  -  May 1st, 2002
(Latest kernel release is 2.5.12)


Features:

Merged
o in 2.5.1+   Rewrite of the block IO (bio) layer             (Jens Axboe)
o in 2.5.2    Initial support for USB 2.0                     (David 
Brownell, Greg Kroah-Hartman, etc.)
o in 2.5.2    Per-process namespaces, late-boot cleanups      (Al Viro, 
Manfred Spraul)
o in 2.5.2+   New scheduler for improved scalability          (Ingo Molnar)
o in 2.5.2+   New kernel device structure (kdev_t)            (Linus 
Torvalds, etc.)
o in 2.5.3    IDE layer update                                (Andre 
Hedrick)
o in 2.5.3    Support reiserfs external journal               (Reiserfs 
team)
o in 2.5.3    Generic ACL (Access Control List) support       (Nathan Scott)
o in 2.5.3    PnP BIOS driver                                 (Alan Cox, 
Thomas Hood, Dave Jones, etc.)
o in 2.5.3+   New driver model & unified device tree          (Patrick 
Mochel)
o in 2.5.4    Add preempt kernel option                       (Robert Love, 
MontaVista team)
o in 2.5.4    Support for Next Generation POSIX Threading     (NGPT team)
o in 2.5.4+   Porting all input devices over to input API     (Vojtech 
Pavlik, James Simmons)
o in 2.5.5    Add ALSA (Advanced Linux Sound Architecture)    (ALSA team)
o in 2.5.5    Pagetables in highmem support                   (Ingo Molnar, 
Arjan van de Ven)
o in 2.5.5    New architecture: AMD 64-bit (x86-64)           (Andi Kleen, 
x86-64 Linux team)
o in 2.5.5    New architecture: PowerPC 64-bit (ppc64)        (Anton 
Blanchard, ppc64 team)
o in 2.5.5+   IDE subsystem major cleanup                     (Martin 
Dalecki, Vojtech Pavlik)
o in 2.5.6    Add JFS (Journaling FileSystem from IBM)        (JFS team)
o in 2.5.6    per_cpu infrastructure                          (Rusty 
Russell)
o in 2.5.6    HDLC (High-level Data Link Control) update      (Krzysztof 
Halasa)
o in 2.5.6    smbfs Unicode and large file support            (Urban 
Widmark) 
o in 2.5.7    New driver API for Wireless Extensions          (Jean 
Tourrilhes)
o in 2.5.7    Video for Linux (V4L) redesign                  (Gerd Knorr)
o in 2.5.7    Futexes (Fast Lightweight Userspace Semaphores) (Rusty 
Russell, etc.)
o in 2.5.7+   NAPI network interrupt mitigation               (Jamal Hadi 
Salim, Robert Olsson, Alexey Kuznetsov)
o in 2.5.7+   ACPI (Advanced Configuration & Power Interface) (Andy Grover, 
ACPI team)
o in 2.5.8    Syscall interface for CPU task affinity         (Robert Love)
o in 2.5.8    Radix-tree pagecache                            (Momchil 
Velikov, Christoph Hellwig)
o in 2.5.8+   Delayed disk block allocation                   (Andrew 
Morton)
o in 2.5.9    Smarter IRQ balancing                           (Ingo Molnar)
o in 2.5.11   Replace old NTFS driver with NTFS TNG driver    (Anton 
Altaparmakov)
o in 2.5.11   Fast walk dcache                                (Hanna Linder)
o in 2.5.11+  Rewrite of the framebuffer layer                (James 
Simmons)

o in -dj      Rewrite of the console layer                    (James 
Simmons)
o in -ac      Strict address space accounting                 (Alan Cox)
o in -ac      PCMCIA Zoom video support                       (Alan Cox)
o in -ac      More complete IEEE 802.2 stack                  (Arnaldo, Jay 
Schullist, from Procom donated code)

o Ready       Better event logging for enterprise systems     (Larry 
Kessler, evlog team)
o Ready       New quota system supporting plugins             (Jan Kara)
o Ready       Linux booting ELF images                        (Eric 
Biederman)
o Ready       First pass at LinuxBIOS support                 (Eric 
Biederman)
o Ready       Build option for Linux Trace Toolkit (LTT)      (Karim 
Yaghmour)

o Beta        New kernel build system (kbuild 2.5)            (Keith Owens)
o Beta        Add support for CPU clock/voltage scaling       (Erik Mouw, 
Dave Jones, Russell King, Arjan van de Ven)
o Beta        Serial driver restructure                       (Russell King)
o Beta        New IO scheduler                                (Jens Axboe)
o Beta        Add XFS (A journaling filesystem from SGI)      (XFS team)
o Beta        New VM with reverse mappings                    (Rik van Riel)
o Beta        Fix long-held locks for low scheduling latency  (Andrew 
Morton, Robert Love, etc.)
o Beta        Add Linux Security Module (LSM)                 (LSM team)
o Beta        Hotplug CPU support                             (Rusty 
Russell)
o Beta        Per-mountpoint read-only, union-mounts, unionfs (Al Viro)
o Beta        EVMS (Enterprise Volume Management System)      (EVMS team)
o Beta        LVM (Logical Volume Manager) v2.0               (LVM team)
o Beta        Dynamic Probes                                  (Suparna 
Bhattacharya, dprobes team)
o Beta        Scalable CPU bitmasks                           (Russ Weight)
o Beta        Page table sharing                              (Daniel 
Phillips)
o Beta        ext2/ext3 online resize support                 (Andreas 
Dilger)
o Beta        Add User-Mode Linux (UML)                       (Jeff Dike)
o Beta        UDF Write support for CD-R/RW (packet writing)  (Jens Axboe, 
Peter Osterlund)
o Beta        Add hardware sensors drivers                    (lm_sensors 
team)
o Beta        New kernel config system: CML2                  (Eric Raymond)
o Beta        Read-Copy Update Mutual Exclusion               (Dipankar 
Sarma, Rusty Russell, Andrea Arcangeli, LSE Team)
o Beta        USB device (not host) support                   (Stuart 
Lynne, Greg Kroah-Hartman)
o Beta        Support for IDE TCQ (Tagged Command Queueing)   (Jens Axboe)

o Alpha       Better support of high-end NUMA machines        (NUMA team)
o Alpha       Add Asynchronous IO (aio) support               (Ben LaHaise)
o Alpha       Overhaul PCMCIA support                         (David 
Woodhouse, David Hinds)
o Alpha       Full compliance with IPv6                       (Alexey 
Kuznetzov, Jun Murai, Yoshifuji Hideaki, USAGI team)
o Alpha       UMSDOS (Unix under MS-DOS) Rewrite              (Al Viro)
o Alpha       Scalable Statistics Counter                     (Ravikiran 
Thirumalai)
o Alpha       Linux Kernel Crash Dumps                        (Matt 
Robinson, LKCD team)
o Alpha       Add support for NFS v4                          (NFS v4 team)
o Alpha       ext2/ext3 large directory support: HTree index  (Daniel 
Phillips, Christopher Li, Ted Ts'o)
o Alpha       Improved i2o (Intelligent Input/Ouput) layer    (Alan Cox)
o Alpha       New MTRR (Memory Type Range Register) driver    (Patrick 
Mochel)
o Alpha       Remove use of the BKL (Big Kernel Lock)         (Alan Cox, 
Robert Love, Neil Brown, Dave Hansen, etc.)
o Alpha       Zerocopy NFS                                    (Hirokazu 
Takahashi)

o Started     More complete NetBEUI stack                     (Arnaldo 
Carvalho de Melo, from Procom donated code)
o Started     Change all drivers to new driver model          (All 
maintainers)
o Started     Reiserfs v4                                     (Reiserfs 
team)
o Started     Move ISDN4Linux to CAPI based interface         (ISDN4Linux 
team)

o Draft #2    New lightweight library (klibc)                 (Greg Kroah-
Hartman)
o Draft #3    Replace initrd by initramfs                     (H. Peter 
Anvin, Al Viro)
o Planning    Add thrashing control                           (Rik van Riel)
o Planning    Remove all hardwired drivers from kernel        (Alan Cox, 
etc.)
o Planning    Generic parameter/command line interface        (Keith Owens)
o Planning    New mount API                                   (Al Viro)
o Planning    Implement new device naming convention          (Device 
naming team)


Cleanups:

Merged
o in 2.5.3    Break Configure.help into multiple files        (Linus 
Torvalds)
o in 2.5.3    Untangle include file dependancies              (Dave Jones, 
Roman Zippel)
o in 2.5.4    Per network protocol slabcache & sock.h         (Arnaldo 
Carvalho de Melo)
o in 2.5.4    Per filesystem slabcache & fs.h                 (Daniel 
Phillips, Jeff Garzik, Al Viro)
o in 2.5.6    Killing kdev_t for block devices                (Al Viro)

o Ready       Switch to ->get_super() for file_system_type    (Al Viro)
o Ready       ->getattr() ->setattr() ->permission() changes  (Al Viro)

o Beta        file.h and INIT_TASK                            (Benjamin 
LaHaise)
o Beta        Proper UFS fixes, ext2 and locking cleanups     (Al Viro)
o Beta        Lifting limitations on mount(2)                 (Al Viro)
o Beta        Remove dcache_lock                              (Maneesh 
Soni, IBM team)

o Alpha       Split up x86 setup.c into managable pieces      (Patrick 
Mochel)

o Started     Reorder x86 initialization                      (Dave Jones, 
Randy Dunlap)

Have some free time and want to help?  Check out the Kernel Janitor 
TO DO list for a list of source code cleanups you can work on.  
A great place to start learning more about kernel internals!

Re: [STATUS 2.5] May 1, 2002 (BKL status)

[Dave Hansen]

Guillaume Boissiere wrote:
> There has also been a lot of work done by various people to 
> remove the BKL from many places, which leads to my question:
> does anyone has a URL with a list of all the places where the
> BKL should eventually be removed and who's working on it?
> 
> It seems like it would be most useful if someone was willing
> to maintain something like this, but it might be a lot of 
> work - I don't know how long the list would be...

I may not be the leading BKL expert, but I play one on TV :)

Perhaps one of the kernel-janitor people would like to assist me with 
this (cc'ing that list).  I'd be willing to keep a web page to list all 
current BKL uses and keep track of them as they are removed/added 
Perhaps a set of web pages which resemble the directory structure of the 
kernel tree would be helpful??

Here's a good question for kernel-janitor, and anyone else who's 
interested, what format describing BKL use would most encourage you to 
go and remove it?  We already have Rick Lindsley's Global spinlock list: 
  http://prdownloads.sourceforge.net/lse/locking_doc-2.4.16 .  The BKL 
use in there is somewhat dated, but might be a good start.

I have some awk scripts that I use on each new kernel release to check 
for new and removed uses of the BKL.  I can adapt these to start 
checking new BK changesets for BKL changes.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [STATUS 2.5] May 1, 2002

[Mike Fedyk]

On Wed, May 01, 2002 at 09:53:37AM -0400, Guillaume Boissiere wrote:
> new framebuffer layer, as well as some more delayed disk block 
> allocation bits.

Actually Andrews work on address_space based writeback is related somewhat,
but really it's a rewrite/cleanup of the buffer layer.  Delayed block
alocation is helped alot by this, and almost depends on it IIRC.

One vote for a seperate listing in the status for "Address Space based
Writeback / Buffer layer cleanup".

Re: [STATUS 2.5] May 1, 2002

[Andrew Morton]

Mike Fedyk wrote:
> 
> On Wed, May 01, 2002 at 09:53:37AM -0400, Guillaume Boissiere wrote:
> > new framebuffer layer, as well as some more delayed disk block
> > allocation bits.
> 
> Actually Andrews work on address_space based writeback is related somewhat,
> but really it's a rewrite/cleanup of the buffer layer.  Delayed block
> alocation is helped alot by this, and almost depends on it IIRC.
> 
> One vote for a seperate listing in the status for "Address Space based
> Writeback / Buffer layer cleanup".

Well the next major step here is going direct
pagecache<->BIO, bypassing the intermediate submit_bh
for most I/O.

Probably that will make most of the performance benefits
of delayed-allocate go away.

There are other reasons for implementing delalloc
(XFS, improved layout, ...).  So it ain't dead yet.


At 48 bytes, 2.5's buffer_head is now precisely half the
size of 2.4's.  I'm hoping to be able to shed another eight
bytes yet.

With the pagecache<->BIO change, the buffer_head will most
definitely become "per-page metadata which describes the state
of sub-page segments" and not "something which is used for
performing I/O".



-

Re: [STATUS 2.5] May 1, 2002

[Stephen Lord]

On Wed, 2002-05-01 at 16:38, Andrew Morton wrote:
> Mike Fedyk wrote:
> > 
> > On Wed, May 01, 2002 at 09:53:37AM -0400, Guillaume Boissiere wrote:
> > > new framebuffer layer, as well as some more delayed disk block
> > > allocation bits.
> > 
> > Actually Andrews work on address_space based writeback is related somewhat,
> > but really it's a rewrite/cleanup of the buffer layer.  Delayed block
> > alocation is helped alot by this, and almost depends on it IIRC.
> > 
> > One vote for a seperate listing in the status for "Address Space based
> > Writeback / Buffer layer cleanup".
> 
> Well the next major step here is going direct
> pagecache<->BIO, bypassing the intermediate submit_bh
> for most I/O.
> 
> Probably that will make most of the performance benefits
> of delayed-allocate go away.

Most of the performance benefits of delayed allocate are that
you do not the hard work of allocating the disk space in each
write call, you get to do it once, in potentially larger chunks,
and often not in the user's context.

Steve

Re: [STATUS 2.5] May 1, 2002

[Eric W. Biederman]

Stephen Lord <lord@sgi.com> writes:

> On Wed, 2002-05-01 at 16:38, Andrew Morton wrote:
> > Mike Fedyk wrote:
> > > 
> > > On Wed, May 01, 2002 at 09:53:37AM -0400, Guillaume Boissiere wrote:
> > > > new framebuffer layer, as well as some more delayed disk block
> > > > allocation bits.
> > > 
> > > Actually Andrews work on address_space based writeback is related somewhat,
> > > but really it's a rewrite/cleanup of the buffer layer.  Delayed block
> > > alocation is helped alot by this, and almost depends on it IIRC.
> > > 
> > > One vote for a seperate listing in the status for "Address Space based
> > > Writeback / Buffer layer cleanup".
> > 
> > Well the next major step here is going direct
> > pagecache<->BIO, bypassing the intermediate submit_bh
> > for most I/O.
> > 
> > Probably that will make most of the performance benefits
> > of delayed-allocate go away.
> 
> Most of the performance benefits of delayed allocate are that
> you do not the hard work of allocating the disk space in each
> write call, you get to do it once, in potentially larger chunks,
> and often not in the user's context.

Except for moving the work out of the users context, ext2 gets
a similar benefit by reserving disk space ahead of time.  So it isn't
clear that you need to have a delayed allocation to achieve this.

Eric

Re: [STATUS 2.5] May 1, 2002

[Stephen Lord]

On Wed, 2002-05-01 at 22:14, Eric W. Biederman wrote:
> Stephen Lord <lord@sgi.com> writes:
> > 
> > Most of the performance benefits of delayed allocate are that
> > you do not the hard work of allocating the disk space in each
> > write call, you get to do it once, in potentially larger chunks,
> > and often not in the user's context.
> 
> Except for moving the work out of the users context, ext2 gets
> a similar benefit by reserving disk space ahead of time.  So it isn't
> clear that you need to have a delayed allocation to achieve this.
> 
> Eric

Doesn't everyone reserve more than the write asks for?

Delayed allocation as implemented in XFS means that during a write
call which has to make more room in a file, a super block counter
is changed, and the extent information is recorded in an in memory
structure associated with the inode.

Yes, if you have all your metadata buffers sitting in memory and are
not journalling changes then all you are doing is a different set of
memory manipulations which are noise on modern cpus when compared with
actually moving the file data out to disk.

Steve

Re: [STATUS 2.5] May 1, 2002 (BKL status)

[Guillaume Boissiere]

> I may not be the leading BKL expert, but I play one on TV :)
> 
> Perhaps one of the kernel-janitor people would like to assist me with 
> this (cc'ing that list).  I'd be willing to keep a web page to list all 
> current BKL uses and keep track of them as they are removed/added 
> Perhaps a set of web pages which resemble the directory structure of the 
> kernel tree would be helpful??
> 
> Here's a good question for kernel-janitor, and anyone else who's 
> interested, what format describing BKL use would most encourage you to 
> go and remove it?  We already have Rick Lindsley's Global spinlock list: 
>   http://prdownloads.sourceforge.net/lse/locking_doc-2.4.16 .  The BKL 
> use in there is somewhat dated, but might be a good start.
> 
> I have some awk scripts that I use on each new kernel release to check 
> for new and removed uses of the BKL.  I can adapt these to start 
> checking new BK changesets for BKL changes.

I'm by no means a BKL expert, not even on TV  :-) , but here is a 
suggestion.
What about a listing sorted by how the BKL should be replaced, 
something like this:

BKL --> dma_spin_lock
- alpha/kernel/ptrace.c  Held while doing a ptrace      (John Doe)
- drivers/block/rd.c  Held while doing whatever         (?)
- ...

BKL --> xprt_lock
- ...

BKL --> no lock required
- ...

BKL instances that will NOT be removed
- ...

The advantage is that it should be easier to group together instances
that require the same type of work.
Comments?

-- Guillaume