ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Hi,

First release of squashfs.  Squashfs is a highly compressed read-only 
filesystem for Linux (kernel 2.4.x).  It uses zlib compression to 
compress both files, inodes and directories.  Inodes in the system are 
very small and all blocks are packed to minimise data overhead. Block 
sizes greater than 4K are supported up to a maximum of 32K.

Squashfs is intended for general read-only filesystem use, for archival 
use, and in embedded systems where low overhead is needed.

Squashfs is available from http://squashfs.sourceforge.net.

The patch file is currently against 2.4.19.  There is further info on 
the filesystem design etc. in the README.

I'l be interested in getting any feedback, advice etc. on it.

Phill Lougher

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Samuel Flory]

Phillip Lougher wrote:

> Hi,
>
> First release of squashfs.  Squashfs is a highly compressed read-only 
> filesystem for Linux (kernel 2.4.x).  It uses zlib compression to 
> compress both files, inodes and directories.  Inodes in the system are 
> very small and all blocks are packed to minimise data overhead. Block 
> sizes greater than 4K are supported up to a maximum of 32K.
>
> Squashfs is intended for general read-only filesystem use, for 
> archival use, and in embedded systems where low overhead is needed.
>
> Squashfs is available from http://squashfs.sourceforge.net.
>
> The patch file is currently against 2.4.19.  There is further info on 
> the filesystem design etc. in the README.
>
> I'l be interested in getting any feedback, advice etc. on it.
>

  What are the advantages of squashfs vs cramfs?

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Samuel Flory wrote:
> Phillip Lougher wrote:
> 
>> Hi,
>>
>> First release of squashfs.  Squashfs is a highly compressed read-only 
>> filesystem for Linux (kernel 2.4.x).  It uses zlib compression to 
>> compress both files, inodes and directories.  Inodes in the system are 
>> very small and all blocks are packed to minimise data overhead. Block 
>> sizes greater than 4K are supported up to a maximum of 32K.
>>
>> Squashfs is intended for general read-only filesystem use, for 
>> archival use, and in embedded systems where low overhead is needed.
>>
>> Squashfs is available from http://squashfs.sourceforge.net.
>>
>> The patch file is currently against 2.4.19.  There is further info on 
>> the filesystem design etc. in the README.
>>
>> I'l be interested in getting any feedback, advice etc. on it.
>>
> 
>  What are the advantages of squashfs vs cramfs?
> 
> 
> 

Cramfs was the inspiration for squashfs.  Squashfs basically gives 
better compression, bigger files/filesystem support, and more inode 
information.

1. Blocks upto 32K are supported - data is compressed in units of 32K 
which achieves better compression ratios than compressing in 4K blocks. 
  Generally using bigger than 4K blocks are a bad idea, because the VFS 
calls the filesystem in 4K pages.  Squashfs explictly pushes the extra 
block data into the page cache.

2. Squashfs compresses inode and directory information in addition to 
file data.  Inodes/directories generally compress down to 50%, or say on 
average 8 bytes or less per inode.

3. All fs data is packed on byte alignments, saving a couple of bytes 
per inode and directory.

4.  Full 32 bit uids/guids are stored (4 bits stored in inode, uses a 
lookup table, to give 48 uids/16 gids).  File sizes upto 2^32 are 
supported.  Timestamp info is stored. Cramfs truncates uids to 16 bits, 
uids to 8 bits.  Cramfs files sizes are upto 2^24.  No timestamp info. 
Squashfs takes advantage of metadata compression to have more info with 
smaller metadata overhead.

5 Symbolic link contents/file indexes are stored inside the inode table,
giving better compression than if they were compressed individually, or 
not compressed.

6. The mksquashfs program doesn't store/mmap the filesystem as it is 
created (it performs file duplicate checking against the partially 
written out compressed fs), and so allows larger filesystems to be created.


Further info on the fs is contained in the README...

Phillip

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Jeff Garzik]

Phillip Lougher wrote:

> Samuel Flory wrote:
>
>> Phillip Lougher wrote:
>>
>>> Hi,
>>>
>>> First release of squashfs.  Squashfs is a highly compressed 
>>> read-only filesystem for Linux (kernel 2.4.x).  It uses zlib 
>>> compression to compress both files, inodes and directories.  Inodes 
>>> in the system are very small and all blocks are packed to minimise 
>>> data overhead. Block sizes greater than 4K are supported up to a 
>>> maximum of 32K.
>>>
>>> Squashfs is intended for general read-only filesystem use, for 
>>> archival use, and in embedded systems where low overhead is needed.
>>>
>>> Squashfs is available from http://squashfs.sourceforge.net.
>>>
>>> The patch file is currently against 2.4.19.  There is further info 
>>> on the filesystem design etc. in the README.
>>>
>>> I'l be interested in getting any feedback, advice etc. on it.
>>>
>>
>>  What are the advantages of squashfs vs cramfs?
>>
>>
>>
>
> Cramfs was the inspiration for squashfs.  Squashfs basically gives 
> better compression, bigger files/filesystem support, and more inode 
> information.
>
> 1. Blocks upto 32K are supported - data is compressed in units of 32K 
> which achieves better compression ratios than compressing in 4K 
> blocks.  Generally using bigger than 4K blocks are a bad idea, because 
> the VFS calls the filesystem in 4K pages.  Squashfs explictly pushes 
> the extra block data into the page cache.

I'm curious if you looked at ntfs-tng's code before implementing this. 
 It's pretty darned optimal...

> 2. Squashfs compresses inode and directory information in addition to 
> file data.  Inodes/directories generally compress down to 50%, or say 
> on average 8 bytes or less per inode.

squashfs or mksquashfs?

A r/w compressed filesystem would be darned useful too :)

    Jeff

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Larry McVoy]

> A r/w compressed filesystem would be darned useful too :)

mmap(2) is, err, hard.  Not impossible, it means the file system has to 
support both compressed and uncompressed files, but it's interesting.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Jeff Garzik]

Larry McVoy wrote:

>>A r/w compressed filesystem would be darned useful too :)
>>    
>>
>
>mmap(2) is, err, hard.
>
The underlying filesystem format can make things easier on you...  and 
given that compressing inevitably requires some amount of data copying, 
it's not terribly difficult.  I wouldn't claim NTFS is anything close to 
well-designed, but supporting compression under Linux on NTFS is at 
least feasible and shouldn't require tons of thought.  (I've looked at 
it when dicking around with NTFS-TNG)

>  Not impossible, it means the file system has to 
>support both compressed and uncompressed files, but it's interesting.
>  
>
well, yeah... ;-)

    Jeff

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Ingo Oeser]

On Wed, Oct 30, 2002 at 03:51:50AM +0000, Phillip Lougher wrote:
> 4.  Full 32 bit uids/guids are stored (4 bits stored in inode, uses a 
> lookup table, to give 48 uids/16 gids).  File sizes upto 2^32 are 
> supported.  Timestamp info is stored. Cramfs truncates uids to 16 bits, 
> uids to 8 bits.  Cramfs files sizes are upto 2^24.  No timestamp info. 
> Squashfs takes advantage of metadata compression to have more info with 
> smaller metadata overhead.

Why limiting to 2GB? AFAIR you wanted to use a cramfs-like
filesystem for backups. Are videos and large data bases not worth
of backing up?

It seems to be good work. So I really wait for Al Viros comments ;-)

Regards

Ingo Oeser
-- 
Science is what we can tell a computer. Art is everything else. --- D.E.Knuth

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Matthew Wilcox]

Larry McVoy wrote:
> > A r/w compressed filesystem would be darned useful too :)
> 
> mmap(2) is, err, hard. Not impossible, it means the file system has to
> support both compressed and uncompressed files, but it's interesting.

Easier than you think, perhaps.  Depends how much compression you're
after, of course, but here's how Acorn did it in RISCiX (a 4.3BSD
derivative):

Pages were 32k (an interesting feature of the MMU...), and the underlying
filesystem was a fairly vanilla BSD FFS (probably 4k blocks with 1k
fragments; discs were around 50MB).  Each page was written at a 32k
boundary, but compressed.  So there were holes in the file where other
files could store their data.  Naturally you waste on average 512 bytes
per 32k page, but I think they managed to get 80MB of unix distro onto
a 50MB disc this way, so it's nothing to be sneezed at.

-- 
Revolutions do not require corporate support.

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Jesse Pollard]

On Tuesday 29 October 2002 10:11 pm, Larry McVoy wrote:
> > A r/w compressed filesystem would be darned useful too :)
>
> mmap(2) is, err, hard.  Not impossible, it means the file system has to
> support both compressed and uncompressed files, but it's interesting.

You can also think of it as a step toward a hierarchical filesystem with the
files:
	1. uncompressed (with uncompressed inode)
	2. compressed on line (real disk space allocated)
	3. compressed nearline (only compressed inode on disk, with a
	reference to offline storage)

Obviously this is only for very large filesystems (we have one FS that
is currently between 100-200 TB in size when you include the migrated
storage).
-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Padraig Brady]

Jesse Pollard wrote:
> On Tuesday 29 October 2002 10:11 pm, Larry McVoy wrote:
> 
>>>A r/w compressed filesystem would be darned useful too :)
>>
>>mmap(2) is, err, hard.  Not impossible, it means the file system has to
>>support both compressed and uncompressed files, but it's interesting.
> 
> You can also think of it as a step toward a hierarchical filesystem with the
> files:
> 	1. uncompressed (with uncompressed inode)
> 	2. compressed on line (real disk space allocated)
> 	3. compressed nearline (only compressed inode on disk, with a
> 	reference to offline storage)
> 
> Obviously this is only for very large filesystems (we have one FS that
> is currently between 100-200 TB in size when you include the migrated
> storage).

I think it's worth referencing e2compr here also,
which is a patch that provides transparent compression
for ext2 (& ext3?). Denis RICHARD (CC'd) is maintaining
the 2.4 implementation. I've version 0.4.42 (against
2.4.16) mirrored here:
http://www.iol.ie/~padraiga/patches/e2compr-0.4.42-patch-2.4.16.gz

Note I've used it without problems (to access filesystems
created with the kernel 2.2 version) here:
http://cvs.bofh.asn.au/e2compr/

Pádraig.

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Denis RICHARD]

Padraig Brady wrote:

> Jesse Pollard wrote:
>
>> On Tuesday 29 October 2002 10:11 pm, Larry McVoy wrote:
>>
>>>> A r/w compressed filesystem would be darned useful too :)
>>>
>>>
>>> mmap(2) is, err, hard.  Not impossible, it means the file system has to
>>> support both compressed and uncompressed files, but it's interesting.
>>
>>
>> You can also think of it as a step toward a hierarchical filesystem 
>> with the
>> files:
>>     1. uncompressed (with uncompressed inode)
>>     2. compressed on line (real disk space allocated)
>>     3. compressed nearline (only compressed inode on disk, with a
>>     reference to offline storage)
>>
>> Obviously this is only for very large filesystems (we have one FS that
>> is currently between 100-200 TB in size when you include the migrated
>> storage).
>
>
> I think it's worth referencing e2compr here also,
> which is a patch that provides transparent compression
> for ext2 (& ext3?). Denis RICHARD (CC'd) is maintaining
> the 2.4 implementation. I've version 0.4.42 (against
> 2.4.16) mirrored here:
> http://www.iol.ie/~padraiga/patches/e2compr-0.4.42-patch-2.4.16.gz 

A new version of the patch (0.4.43) is available here :
http://www.alizt.com
But I have not tested the patch on ext3.

>
>
> Note I've used it without problems (to access filesystems
> created with the kernel 2.2 version) here:
> http://cvs.bofh.asn.au/e2compr/
>
> Pádraig.
>
>
-- 
-----------------------------\--------------------------\
Denis RICHARD                 \ ALCATEL Business Systems \
mailto:dri@sxb.bsf.alcatel.fr / Tel: +33(0)3 90 67 69 36 /
-----------------------------/--------------------------/

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Jeff Garzik wrote:
 > Phillip Lougher wrote:

 >
 >> 2. Squashfs compresses inode and directory information in addition
 >> to file data.  Inodes/directories generally compress down to 50%, or
 >> say on average 8 bytes or less per inode.
 >
 >
 > squashfs or mksquashfs?

mksquashfs...

 >
 > A r/w compressed filesystem would be darned useful too :)
 >
 > Jeff
 >

A r/w compressed filesystem may be my next project...  As a couple of
people have mentioned there are compressed r/w filesystems already
out there.

As you'll know, there are always tradeoffs with filesystem design,
it is very difficult to get as good compression with a r/w fs
than with a read only filesystem.  I wanted to get maximum
compression, and quite a few of the techniques I use rely on
its read-only nature.

An append only (i.e. files can be added, but not modified), fs might
be a useful compromise.  With compressed metadata, any modification
of files will inevitably achive different compression ratios, and so
modification of metadata/files in place is not an option.  Appending
modified metadata/data brings you to log-structured (journalling)
filesystems and compaction (log cleaning) requirements with consequent
loss of compression.

Phillip


 >
 >
 >
 >

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Ingo Oeser wrote:
> On Wed, Oct 30, 2002 at 03:51:50AM +0000, Phillip Lougher wrote:
>
>>File sizes upto 2^32 are supported.
 >
> Why limiting to 2GB? AFAIR you wanted to use a cramfs-like
> filesystem for backups. Are videos and large data bases not worth
> of backing up?
> 

Why are files limited to 4GB? (2^32). Simply because I never
thought anything bigger was needed :-)  Initially files were limited to
2^24 (like cramfs), but because of compressed metadata I thought I could
justify an extra 8 bits in the inode, in consideration of the
extra functionality.

Going from 32 bits to 40 bits say is a bigger jump - an extra
2 bytes per file inode (1 byte for the extra file size, and 1
byte because the file start block pointer must increase
by 1 byte as well).  Considering the fact I've tried to squeeze
every last byte out...

However, I could add the extra two bytes if people thought
it was worth it.

Alternatively,  squashfs uses different inodes per file type.  I could
add an extra "big file" type to deal with files bigger than 4GB.  This
would mean > 4GB files are supported, with only one extra byte per inode
for smaller files.  I'll think about it...

> It seems to be good work. So I really wait for Al Viros comments ;-)

Thanks!

Phillip

> Regards
> 
> Ingo Oeser

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Matthew Wilcox wrote:
 >Larry McVoy wrote:
 >> > A r/w compressed filesystem would be darned useful too :)
 >> > mmap(2) is, err, hard. Not impossible, it means the file system 
has >to
 >> support both compressed and uncompressed files, but it's interesting.
 >
 >Easier than you think, perhaps.  Depends how much compression you're
 >after, of course, but here's how Acorn did it in RISCiX (a 4.3BSD
 >derivative):
 >

Strange you should mention this... I used to work at Acorn with
the guy who did this (Mark Taunton). RISCiX was too early to be 4.3, I
think it was 4.1, but may have been 4.2, certainly no later, this was
1989.

 >Pages were 32k (an interesting feature of the MMU...), and the 
 >underlying filesystem was a fairly vanilla BSD FFS (probably 4k blocks
 >with 1k fragments; discs were around 50MB).  Each page was written at 
a >32k boundary, but compressed.  So there were holes in the file where 
 >other files could store their data.  Naturally you waste on average 
512 >bytes per 32k page, but I think they managed to get 80MB of unix 
distro >onto a 50MB disc this way, so it's nothing to be sneezed at.

They only needed to compress the fs on the R140 (an ARM 2 based
machine).  The better R260 had a bigger disk, and so didn't need the
compression. Interestingly, because of the slow disk I/O, compression
made the filesystem access faster.

I had an R140, and space was so tight in the filesystem, you had
to delete part of the distribution before you could even compile
a program!

Phillip

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Nicholas Wourms]

Denis RICHARD wrote:

> 
> A new version of the patch (0.4.43) is available here :
> http://www.alizt.com
> But I have not tested the patch on ext3.
> 

Any possibility for a forward port to 2.5?

Cheers,
Nicholas

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Rob Landley]

On Wednesday 30 October 2002 23:56, Phillip Lougher wrote:
> Jeff Garzik wrote:
>  > Phillip Lougher wrote:
>  >> 2. Squashfs compresses inode and directory information in addition
>  >> to file data.  Inodes/directories generally compress down to 50%, or
>  >> say on average 8 bytes or less per inode.
>  >
>  > squashfs or mksquashfs?
>
> mksquashfs...
>
>  > A r/w compressed filesystem would be darned useful too :)
>  >
>  > Jeff
>
> A r/w compressed filesystem may be my next project...  As a couple of
> people have mentioned there are compressed r/w filesystems already
> out there.
>
> As you'll know, there are always tradeoffs with filesystem design,
> it is very difficult to get as good compression with a r/w fs
> than with a read only filesystem.  I wanted to get maximum
> compression, and quite a few of the techniques I use rely on
> its read-only nature.

A compressed filesystem with dynamically updated random-access files will 
fragment the heck out of itself darn fast.  (Seek into the middle of a file 
somewhere, write a block, seek somewhere else, write another block, repeat 
1000 times...  Keep in mind that the new compressed block of data will almost 
certainly not be the same size as the old one... It's a mess.)

My potential usage is: I've got a little linux distribution I put together 
called "firmware linux", which builds from source and outputs a zisofs image 
that gets loopback mounted as the root filesystem.  (A very alpha version 
could be sucked off of http://216.143.22.141/firmware/fwl-0.8.tar, edit 
"build.sh" to specify the output directory, and then run it.  The point is, 
the whole OS and all applications can be upgraded as one file.  No package 
management, it's basically a big firmware image.)

The reason I used a zisofs instead of cramfs is that cramfs has a LOT of 
problems with big filesystems.  (The finished root partition, with apache and 
samba and ntop and python and rsync and openssh and everything, is currently 
around 100 megs.  Yeah, I can trim that down by quite a bit if I get time, 
I'm currently compilling and developing it under itself so I have gcc in 
there and the full set of man pages and everything...)

Mkcramfs seems to barf at somewhere around 16 megs, which is really limiting.  
AND it seems to try to open every file simultaneously (hardlink detection?) 
so it runs out of file handles.  (Again, that could be adjusted under /proc 
somewhere, but isn't worth it.)

So it seems that the thing to test this against isn't cramfs, but zisofs.  
Have you looked at that?

I'll take a look myself and get back to you...

Rob

-- 
http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, 
CmdrTaco, liquid nitrogen ice cream, and caffienated jello.  Well why not?

Re: ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)

[Phillip Lougher]

Rob Landley wrote:
> On Wednesday 30 October 2002 23:56, Phillip Lougher wrote:
> 
>>Jeff Garzik wrote:
>> > Phillip Lougher wrote:
>> >> 2. Squashfs compresses inode and directory information in addition
>> >> to file data.  Inodes/directories generally compress down to 50%, or
>> >> say on average 8 bytes or less per inode.
>> >
>> > squashfs or mksquashfs?
>>
>>mksquashfs...
>>
>> > A r/w compressed filesystem would be darned useful too :)
>> >
>> > Jeff
>>
>>A r/w compressed filesystem may be my next project...  As a couple of
>>people have mentioned there are compressed r/w filesystems already
>>out there.
>>
>>As you'll know, there are always tradeoffs with filesystem design,
>>it is very difficult to get as good compression with a r/w fs
>>than with a read only filesystem.  I wanted to get maximum
>>compression, and quite a few of the techniques I use rely on
>>its read-only nature.
> 
> 
> A compressed filesystem with dynamically updated random-access files will 
> fragment the heck out of itself darn fast.  (Seek into the middle of a file 
> somewhere, write a block, seek somewhere else, write another block, repeat 
> 1000 times...  Keep in mind that the new compressed block of data will almost 
> certainly not be the same size as the old one... It's a mess.)
> 
> My potential usage is: I've got a little linux distribution I put together 
> called "firmware linux", which builds from source and outputs a zisofs image 
> that gets loopback mounted as the root filesystem.  (A very alpha version 
> could be sucked off of http://216.143.22.141/firmware/fwl-0.8.tar, edit 
> "build.sh" to specify the output directory, and then run it.  The point is, 
> the whole OS and all applications can be upgraded as one file.  No package 
> management, it's basically a big firmware image.)
> 
> The reason I used a zisofs instead of cramfs is that cramfs has a LOT of 
> problems with big filesystems.  (The finished root partition, with apache and 
> samba and ntop and python and rsync and openssh and everything, is currently 
> around 100 megs.  Yeah, I can trim that down by quite a bit if I get time, 
> I'm currently compilling and developing it under itself so I have gcc in 
> there and the full set of man pages and everything...)
> 
> Mkcramfs seems to barf at somewhere around 16 megs, which is really limiting.  
> AND it seems to try to open every file simultaneously (hardlink detection?) 
> so it runs out of file handles.  (Again, that could be adjusted under /proc 
> somewhere, but isn't worth it.)
> 
> So it seems that the thing to test this against isn't cramfs, but zisofs.  
> Have you looked at that?
> 
> I'll take a look myself and get back to you...
> 
> Rob
> 

Hi,

I looked at both cramfs and ziosfs when writing squashfs.  Zisofs is a 
nice fs, but tends to have greater overhead due to the isofs filesystem. 
  On tests I've found zisofs images to be between 5% (a single directory 
with lots of small binaries) and 61% (lots of nested directories) bigger 
than the squashfs filesystem.

I believe that squashfs is useful for what you're doing.  I'm a bit 
hesitant in saying that, because I'd rather people downloaded it and 
made up their own minds :-)

Thank you for trying it out, and I hope you like it.  I'll obviously be 
interested in your thoughts on it.

Phillip