[ANN] Squashfs 3.3 released

[ANN] Squashfs 3.3 released

[Phillip Lougher]

Hi,

I'm pleased to announce another release of Squashfs.  This is the 22nd
release in just over five years.  Squashfs 3.3 has lots of nice improvements,
both to the filesystem itself (bigger blocks and sparse files), but
also to the Squashfs-tools Mksquashfs and Unsquashfs.

The next stage after this release is to fix the one remaining blocking issue
(filesystem endianness), and then try to get Squashfs mainlined into the
Linux kernel again.

The list of changes from the change-log are as follows:

1. Filesystem improvements:

    1.1. Maximum block size has been increased to 1Mbyte, and the
         default block size has been increased to 128 Kbytes.
         This improves compression.

    1.2. Sparse files are now supported.  Sparse files are files
         which have large areas of unallocated data commonly called
         holes.  These files are now detected by Squashfs and stored
         more efficiently.  This improves compression and read
         performance for sparse files.

2. Mksquashfs improvements:

    2.1.  Exclude files have been extended to use wildcard pattern
          matching and regular expressions.  Support has also been
          added for non-anchored excludes, which means it is
          now possible to specify excludes which match anywhere
          in the filesystem (i.e. leaf files), rather than always
          having to specify exclude files starting from the root
          directory (anchored excludes).

    2.2.  Recovery files are now created when appending to existing
          Squashfs filesystems.  This allows the original filesystem
          to be recovered if Mksquashfs aborts unexpectedly
          (i.e. power failure).

3. Unsquashfs improvements:

     3.1. Multiple extract files can now be specified on the
          command line, and the files/directories to be extracted can
          now also be given in a file.

     3.2. Extract files have been extended to use wildcard pattern
          matching and regular expressions.

     3.3. Filename printing has been enhanced and Unquashfs can
          now display filenames with file attributes
          ('ls -l' style output).

     3.4. A -stat option has been added which displays the filesystem
          superblock information.

     3.5. Unsquashfs now supports 1.x filesystems.

4. Miscellaneous improvements/bug fixes:

    4.1. Squashfs kernel code improved to use SetPageError in
         squashfs_readpage() if I/O error occurs.

    4.2. Fixed Squashfs kernel code bug preventing file
         seeking beyond 2GB.

    4.3. Mksquashfs now detects file size changes between
         first phase directory scan and second phase filesystem create.

Regards

Phillip Lougher

Re: [ANN] Squashfs 3.3 released

[maximilian attems]

On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
> I'm pleased to announce another release of Squashfs.  This is the 22nd
> release in just over five years.  Squashfs 3.3 has lots of nice 
> improvements,
> both to the filesystem itself (bigger blocks and sparse files), but
> also to the Squashfs-tools Mksquashfs and Unsquashfs.
> 
> The next stage after this release is to fix the one remaining blocking issue
> (filesystem endianness), and then try to get Squashfs mainlined into the
> Linux kernel again.
> 

that would be very cool!
with my hat as debian kernel maintainer i'd be very relieved to see it
mainlined. i don't know of any major distro that doesn't ship it.

thanks

-- 
maks

Re: [ANN] Squashfs 3.3 released

[Michael Tokarev]

Phillip Lougher wrote:
> Hi,
> 
> I'm pleased to announce another release of Squashfs.  This is the 22nd
> release in just over five years.

Thanks Phillip.

A tiny bug[fix] I always forgot to send...  In fs/squashfs/inode.c,
constants TASK_UNINTERRUPTIBLE and TASK_INTERRUPTIBLE are used, but
they aren't sometimes defined (declared in linux/sched.h):

  CC [M]  fs/squashfs/inode.o
fs/squashfs/inode.c: In function 'squashfs_get_cached_block':
fs/squashfs/inode.c:367: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:367: error: (Each undeclared identifier is reported only once
fs/squashfs/inode.c:367: error: for each function it appears in.)
fs/squashfs/inode.c:367: warning: implicit declaration of function 'schedule'
fs/squashfs/inode.c:404: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c: In function 'release_cached_fragment':
fs/squashfs/inode.c:499: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:499: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c: In function 'get_cached_fragment':
fs/squashfs/inode.c:522: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:559: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)

I'm not exactly sure which config option is "at blame"
(this is an i486-based UP generic-hardware config), but
I'm not interested to know, either.  The following
trivial patch fixes it once for all.

--- linux-2.6.22.orig/fs/squashfs/inode.c       2007-07-12 14:57:22.000000000 +0400
+++ linux-2.6.22/fs/squashfs/inode.c    2007-07-12 14:57:53.000000000 +0400
@@ -31,6 +31,7 @@
 #include <linux/vfs.h>
 #include <linux/vmalloc.h>
+#include <linux/sched.h>
 #include <linux/smp_lock.h>

 #include "squashfs.h"


It was needed for v3.2 too.

Thanks.

/mjt

Re: [ANN] Squashfs 3.3 released

[Phillip Lougher]

Michael Tokarev wrote:

> 
> A tiny bug[fix] I always forgot to send...  In fs/squashfs/inode.c,
> constants TASK_UNINTERRUPTIBLE and TASK_INTERRUPTIBLE are used, but
> they aren't sometimes defined (declared in linux/sched.h):
> 

Thanks - Squashfs gained a lot of #includes over time, many which I deemed were
unnecessary and removed in Squashfs 3.2.   I obviously removed too many.
Fix applied to CVS.

Phillip

Re: [ANN] Squashfs 3.3 released

[Phillip Lougher]

maximilian attems wrote:
> On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
>
>> The next stage after this release is to fix the one remaining blocking issue
>> (filesystem endianness), and then try to get Squashfs mainlined into the
>> Linux kernel again.
>>
> 
> that would be very cool!

Yes, it would be cool :)  Five years is a long time to maintain
something out of tree, especially recently when there's been
so many minor changes to the VFS interface between kernel releases.

> with my hat as debian kernel maintainer i'd be very relieved to see it
> mainlined. i don't know of any major distro that doesn't ship it.
> 

I don't know of any major distro that doesn't ship Squashfs either
(except arguably Slackware).  Putting my other hat on (one of the
Ubuntu kernel maintainers) I don't think Squashfs has caused
distros that many problems because it is an easy patch to apply
(it doesn't touch that many kernel files), but it is always good
to minimise the differences from the stock kernel.org kernel.

Phillip

Re: [ANN] Squashfs 3.3 released

[Dave Jones]

On Wed, Nov 07, 2007 at 04:32:00PM +0000, Phillip Lougher wrote:
 > maximilian attems wrote:
 > > On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
 > >
 > >> The next stage after this release is to fix the one remaining blocking issue
 > >> (filesystem endianness), and then try to get Squashfs mainlined into the
 > >> Linux kernel again.
 > >>
 > > 
 > > that would be very cool!
 > 
 > Yes, it would be cool :)  Five years is a long time to maintain
 > something out of tree, especially recently when there's been
 > so many minor changes to the VFS interface between kernel releases.
 > 
 > > with my hat as debian kernel maintainer i'd be very relieved to see it
 > > mainlined. i don't know of any major distro that doesn't ship it.
 > > 
 > 
 > I don't know of any major distro that doesn't ship Squashfs either
 > (except arguably Slackware).  Putting my other hat on (one of the
 > Ubuntu kernel maintainers) I don't think Squashfs has caused
 > distros that many problems because it is an easy patch to apply
 > (it doesn't touch that many kernel files), but it is always good
 > to minimise the differences from the stock kernel.org kernel.

The biggest problem we've seen with it (asides from having to rediff
it every time we rebase when there isn't a newer upstream) is complaints
along the lines of "my Fedora 7 kernel can't unpack squashfs images
from Fedora 5"
(s/Fedora 5/other random older distros/ )

If the format is now stable however, it would be great to get it upstream.

	Dave

-- 
http://www.codemonkey.org.uk

Re: [ANN] Squashfs 3.3 released

[Phillip Lougher]

Dave Jones wrote:

> 
> The biggest problem we've seen with it (asides from having to rediff
> it every time we rebase when there isn't a newer upstream)

Yes, this is mainly my fault.  There was a gap of 10 months between the 
3.2 release in January this year, and the latest in November. With the 
rate of new kernel releases this wasn't really acceptable because the 
January release was stuck with a patch for kernels no newer than 2.6.20. 
I received numerous complaints about it.

Some of you may be aware that I started work at Canonical, and this left 
almost no spare-time to work on Squashfs for 9 months.

>is complaints
> along the lines of "my Fedora 7 kernel can't unpack squashfs images
> from Fedora 5"
> (s/Fedora 5/other random older distros/ )
>

Squashfs has backwards compatibility with older versions, and it should 
mount all older versions back to 2.0 (released May 2004). Unfortunately 
RedHat grabbed a CVS version of Squashfs just before the 3.0 release. 
This was development code, and release testing showed it
had a bug where it couldn't mount older versions. It was fixed for
release.

> If the format is now stable however, it would be great to get it upstream.
> 

The move from the 2.0 format to the later 3.0 format was mainly forced 
by the demands of the Linux kernel mailing list when I first submitted 
it early 2005.  There was no other way to incorporate demands for larger 
than 4GB filesystems, and provide support for "." and ".." in readdir 
without modifying the filesystem format.

Unfortunately the move to fixed little endian filesystem will involve 
another filesystem layout change.  The current filesystem layout still 
uses packed bitfield structures, and it is impossible to swap these 
using the standard kernel swap macros.  Removal of my routines that can 
properly swap packed bitfield structures is another change demanded by 
the Linux kernel mailing list.

Once the little endian work has been done, and hopefully once it is in 
the kernel, I don't anticipate any further layout changes.

Phillip

Re: [ANN] Squashfs 3.3 released

[Christoph Hellwig]

On Wed, Nov 21, 2007 at 02:02:43PM +0000, Phillip Lougher wrote:
> Unfortunately the move to fixed little endian filesystem will involve 
> another filesystem layout change.  The current filesystem layout still 
> uses packed bitfield structures, and it is impossible to swap these 
> using the standard kernel swap macros.  Removal of my routines that can 
> properly swap packed bitfield structures is another change demanded by 
> the Linux kernel mailing list.

The normal way to do it is to use shift and mask after doing the endian
conversion.  But the problem with bitfields is that they can have different
kinds of layouts depending on the compiler or abi which is another reason
to avoid them in ondisk/wire formats.

Re: [ANN] Squashfs 3.3 released

[Phillip Lougher]

Christoph Hellwig wrote:
> On Wed, Nov 21, 2007 at 02:02:43PM +0000, Phillip Lougher wrote:
>> Unfortunately the move to fixed little endian filesystem will involve 
>> another filesystem layout change.  The current filesystem layout still 
>> uses packed bitfield structures, and it is impossible to swap these 
>> using the standard kernel swap macros.  Removal of my routines that can 
>> properly swap packed bitfield structures is another change demanded by 
>> the Linux kernel mailing list.
> 
> The normal way to do it is to use shift and mask after doing the endian
> conversion.  But the problem with bitfields is that they can have different
> kinds of layouts depending on the compiler or abi which is another reason
> to avoid them in ondisk/wire formats.
> 

Yes, the bitfields are packed differently on little and big endian 
architectures which mean they appear in different places in the 
structure. I want to move away from that mess when I move to little 
endian only.

Phillip