Squashfs 4.0 tools

Squashfs 4.0 tools

[Stephen Hemminger]

It is great that squashfs is finally in mainline kernel. But the
squashfs tools are tightly bound to the version of the squashfs filesystem.
The latest tools in Debian (and the download page) are limited to 
version 3.4 but the upstream kernel version is 4.0!

Do we have a cart with no horse here?

Re: Squashfs 4.0 tools

[Phillip Lougher]

Stephen Hemminger wrote:
> It is great that squashfs is finally in mainline kernel. But the
> squashfs tools are tightly bound to the version of the squashfs filesystem.
> The latest tools in Debian (and the download page) are limited to 
> version 3.4 but the upstream kernel version is 4.0!
> 
> Do we have a cart with no horse here?
> 

The horse is currently in training, or to put it more mundanely, the
squashfs 4.0 tools are currently in Squashfs CVS :-)

I have been working on the 4.0 squashfs tools for the last couple of
months making them (mksquashfs and unsquashfs) ready for a supported
release.  Now that Linux 2.6.29 has been released (3 hours ago), I
need to make this release ASAP, which as everything is almost finished,
should be sometime this week.

Phillip

Re: Squashfs 4.0 tools

[Michael Tokarev]

Phillip Lougher wrote:
> Stephen Hemminger wrote:
>> It is great that squashfs is finally in mainline kernel. But the
>> squashfs tools are tightly bound to the version of the squashfs 
>> filesystem.
[]
> I have been working on the 4.0 squashfs tools for the last couple of
> months making them (mksquashfs and unsquashfs) ready for a supported
> release.  Now that Linux 2.6.29 has been released (3 hours ago), I
> need to make this release ASAP, which as everything is almost finished,
> should be sometime this week.

By the way, with this on-disk format change, is there a way to get
4.0 fs code working with 2.6.27 (long-time-stable) kernel?  I tried
to compile it on 2.6.27 but it fails due to some API changes and I'm
not that deep into kernel to fix it...

Thanks!

Re: [Squashfs-devel] Squashfs 4.0 tools

[Phillip Lougher]

Michael Tokarev wrote:
> Phillip Lougher wrote:
>> Stephen Hemminger wrote:
>>> It is great that squashfs is finally in mainline kernel. But the
>>> squashfs tools are tightly bound to the version of the squashfs 
>>> filesystem.
> []
>> I have been working on the 4.0 squashfs tools for the last couple of
>> months making them (mksquashfs and unsquashfs) ready for a supported
>> release.  Now that Linux 2.6.29 has been released (3 hours ago), I
>> need to make this release ASAP, which as everything is almost finished,
>> should be sometime this week.
> 
> By the way, with this on-disk format change, is there a way to get
> 4.0 fs code working with 2.6.27 (long-time-stable) kernel?  
I tried  to compile it on 2.6.27 but it fails due to some API changes and I'm
> not that deep into kernel to fix it...
> 

Yes I know.  The 4.0 patches only work on 2.6.28 or later AFAIK.

When I get time and if there's demand (you're the first person to ask
and so there's obviously some demand), I'll backport the patches
to earlier kernels.

Phillip

Re: Squashfs 4.0 tools

[Rob Landley]

On Monday 23 March 2009 21:39:02 Phillip Lougher wrote:
> Stephen Hemminger wrote:
> > It is great that squashfs is finally in mainline kernel. But the
> > squashfs tools are tightly bound to the version of the squashfs
> > filesystem. The latest tools in Debian (and the download page) are
> > limited to version 3.4 but the upstream kernel version is 4.0!
> >
> > Do we have a cart with no horse here?
>
> The horse is currently in training, or to put it more mundanely, the
> squashfs 4.0 tools are currently in Squashfs CVS :-)
>
> I have been working on the 4.0 squashfs tools for the last couple of
> months making them (mksquashfs and unsquashfs) ready for a supported
> release.  Now that Linux 2.6.29 has been released (3 hours ago), I
> need to make this release ASAP, which as everything is almost finished,
> should be sometime this week.

Any updated ETA on this?

Rob

P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to 
it from the squashfs.org page.  Buried in the middle of a lot of text is a 
link to the sourceforge squashfs project page, which has a link to "forums" 
which are not the same as mailing lists.

Since I knew it's called "squashfs-devel" I could google for that and find 
http://sourceforge.net/mailarchive/forum.php?forum_name=squashfs-devel

But it might be nice to at least _hint_ about its existence on the web page.  
The main page's "contact" section is your personal email and nothing else...

Re: Squashfs 4.0 tools

[Geert Uytterhoeven]

On Sun, Mar 29, 2009 at 00:27, Rob Landley <rob@landley.net> wrote:
> P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to
> it from the squashfs.org page.  Buried in the middle of a lot of text is a
> link to the sourceforge squashfs project page, which has a link to "forums"
> which are not the same as mailing lists.

There should be a link to it somewhere on the SF project page...

> But it might be nice to at least _hint_ about its existence on the web page.
> The main page's "contact" section is your personal email and nothing else...

Yep. Fortunately we also have MAINTAINERS ;-)

| SQUASHFS FILE SYSTEM
| P:      Phillip Lougher
| M:      phillip@lougher.demon.co.uk
| L:      squashfs-devel@lists.sourceforge.net (subscribers-only)
| W:      http://squashfs.org.uk
| S:      Maintained

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Squashfs 4.0 tools

[Stephen Hemminger]

On Sun, 29 Mar 2009 11:02:19 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> On Sun, Mar 29, 2009 at 00:27, Rob Landley <rob@landley.net> wrote:
> > P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to
> > it from the squashfs.org page.  Buried in the middle of a lot of text is a
> > link to the sourceforge squashfs project page, which has a link to "forums"
> > which are not the same as mailing lists.
> 
> There should be a link to it somewhere on the SF project page...
> 
> > But it might be nice to at least _hint_ about its existence on the web page.
> > The main page's "contact" section is your personal email and nothing else...
> 
> Yep. Fortunately we also have MAINTAINERS ;-)
> 
> | SQUASHFS FILE SYSTEM
> | P:      Phillip Lougher
> | M:      phillip@lougher.demon.co.uk
> | L:      squashfs-devel@lists.sourceforge.net (subscribers-only)
> | W:      http://squashfs.org.uk
> | S:      Maintained
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

I built it from cvs (imported into git), and it works okay.

Re: Squashfs 4.0 tools

[Phillip Lougher]

Stephen Hemminger wrote:
> On Sun, 29 Mar 2009 11:02:19 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> 
>> On Sun, Mar 29, 2009 at 00:27, Rob Landley <rob@landley.net> wrote:
>>> P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to
>>> it from the squashfs.org page.  Buried in the middle of a lot of text is a
>>> link to the sourceforge squashfs project page, which has a link to "forums"
>>> which are not the same as mailing lists.
>> There should be a link to it somewhere on the SF project page...
>>
>>> But it might be nice to at least _hint_ about its existence on the web page.
>>> The main page's "contact" section is your personal email and nothing else...

Squashfs is a sourceforge project, and it has a link to the main sourceforge
page from the webpage.  I've always thought that sufficient because users
can get to the mailing lists, web forum, bug tracker etc. from that, and then
choose what they think is the most appropriate way to report issues.

Missing information is simply that, and I rely on feedback from users to
say if I've forgotten to add important or relevant information anywhere.
It would be helpful if that feedback didn't include an implied
criticism, that way we'll all get along better.

> 
> I built it from cvs (imported into git), and it works okay.
> 

Good, this is what I suggested you do :-)  CVS contains everything I've
been working on for the last couple of months.

I intended to do a release last week, but, release testing found a number
of bugs and other issues which had to be addressed, and which consequently
delayed the release.

If you grabbed the tools from CVS more than a day ago you'll find
there's problems with Unsquashfs and big-endian systems.  The last
bug fix (a deadlock issue in mksquashfs undiscovered since 3.4)
went in just now, and so CVS should be what's in the release.

BTW Stephen, it would be helpful if you subscribed to the
squashfs-devel mailing list, at the moment I've having to
authorise all your posts which is a bit of a pain.

Phillip

Re: Squashfs 4.0 tools

[Rob Landley]

On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:
> I built it from cvs (imported into git), and it works okay.

I grabbed the cvs snapshot gentoo's been using, but I don't see any way to 
feed in a device list text file so I can create the /dev/console node root 
filesystems need without needing root access on the host to mknod.

genext2fs has the -D option, which takes:

  /dev d 755 0 0 - - - - -
  /dev/console c 640 0 0 5 1 0 0 -

gen_init_cpio takes:

  dir /dev 755 0 0
  nod /dev/console 660 0 0 c 5 1

Is there a squashfs option I'm missing...?

Rob
-- 
GPLv3 is to GPLv2 what Attack of the Clones is to The Empire Strikes Back.

Re: Squashfs 4.0 tools

[Phillip Lougher]

Rob Landley wrote:
> On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:
>> I built it from cvs (imported into git), and it works okay.
> 
> I grabbed the cvs snapshot gentoo's been using, but I don't see any way to 
> feed in a device list text file so I can create the /dev/console node root 
> filesystems need without needing root access on the host to mknod.
> 
> genext2fs has the -D option, which takes:
> 
>   /dev d 755 0 0 - - - - -
>   /dev/console c 640 0 0 5 1 0 0 -
> 
> gen_init_cpio takes:
> 
>   dir /dev 755 0 0
>   nod /dev/console 660 0 0 c 5 1
> 
> Is there a squashfs option I'm missing...?
> 

No you're unfortunately not missing an option :-)  That option
doesn't exist.

But thanks for sending the email, a couple of people over the
years have asked for something like this, and you've reminded
me that this option doesn't exist in Mksquashfs...

I have thought about adding this option a couple of times, but
it has always been dropped through lack of time.  Efficiently
adding the option isn't as easy as it sounds because the "fake"
device nodes have to somehow be made available to the
recursive directory scan routine that scans the source
directories for files.  Adding a list of fake device
nodes which is scanned on every directory for potential inclusion
in that directory is easy, but not terribly efficient.

I'll delay the release of the 4.0 tools and see if I can
implement this option in the next day or two.

Phillip

Re: [Squashfs-devel] Squashfs 4.0 tools

[Peter Korsgaard]

>>>>> "Phillip" == Phillip Lougher <phillip@lougher.demon.co.uk> writes:

Hi,

 Phillip> I have thought about adding this option a couple of times,
 Phillip> but it has always been dropped through lack of time.
 Phillip> Efficiently adding the option isn't as easy as it sounds
 Phillip> because the "fake" device nodes have to somehow be made
 Phillip> available to the recursive directory scan routine that scans
 Phillip> the source directories for files.  Adding a list of fake
 Phillip> device nodes which is scanned on every directory for
 Phillip> potential inclusion in that directory is easy, but not
 Phillip> terribly efficient.

The normal solution for this is fakeroot, which is also what we're
using for mksquashfs in buildroot:

http://git.buildroot.net/buildroot/tree/target/squashfs/squashfsroot.mk

 Phillip> I'll delay the release of the 4.0 tools and see if I can
 Phillip> implement this option in the next day or two.

I would say rather get 4.0 out of the door as soon as possible and
wait with such nice-to-have stuff for later.

-- 
Bye, Peter Korsgaard

Re: Squashfs 4.0 tools

[Phillip Lougher]

Phillip Lougher wrote:
> Rob Landley wrote:
>> On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:
>>> I built it from cvs (imported into git), and it works okay.
>>
>> I grabbed the cvs snapshot gentoo's been using, but I don't see any 
>> way to feed in a device list text file so I can create the 
>> /dev/console node root filesystems need without needing root access on 
>> the host to mknod.
>>
>> genext2fs has the -D option, which takes:
>>
>>   /dev d 755 0 0 - - - - -
>>   /dev/console c 640 0 0 5 1 0 0 -
>>
>> gen_init_cpio takes:
>>
>>   dir /dev 755 0 0
>>   nod /dev/console 660 0 0 c 5 1
>>
>> Is there a squashfs option I'm missing...?
>>
> I'll delay the release of the 4.0 tools and see if I can
> implement this option in the next day or two.

The Squashfs 4.0 tools have been released on sourceforge
(http://www.squashfs.org), and mksquashfs now includes support for
fake devices and directories.

 From README-4.0

Mksquashfs now supports pseudo files, these allow fake directories, character
and block devices to be specified and added to the Squashfs filesystem being
built, rather than requiring them to be present in the source directories.
This, for example, allows device nodes to be added to the filesystem without
requiring root access.

Two options are supported, -p allows one pseudo file to be specified on the
command line, and -pf allows a pseudo file to be specified containing a
list of pseudo definitions, one per line.

Pseudo device nodes are specified using 7 arguments

Filename type mode uid gid major minor

Where type is either
         b - for block devices, and
         c - for character devices

mode is the octal mode specifier, similar to that expected by chmod.

Uid and gid can be either specified as a decimal number, or by name.

For example:

/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 444 0 0 200 200

Directories are specified using 5 arguments

Filename type mode uid gid

Where type is d.

To ensure the pseudo definition is parsed as one argument and to
ensure it isn't interpreted by the shell, the definition should be enclosed
in quotes when using the -p option, i.e.

mksquashfs test test.sqsh -p "/dev/chrdev c 666 root 100 1"

If a directory given in a pseudo file path exists in the real source
directories it doesn't have to be specified as a pseudo directory, if it
doesn't exist in the source directories, it must be specified as a pseudo
directory (otherwise any pseudo devices under the directory will be ignored).

For example

/dev d 777 root root
/dev/chr_dev c 666 root root 100 1

Phillip

Re: Squashfs 4.0 tools

[Rob Landley]

On Sunday 05 April 2009 18:27:49 Phillip Lougher wrote:
> Phillip Lougher wrote:
> > Rob Landley wrote:
> >> On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:
> >>> I built it from cvs (imported into git), and it works okay.
> >>
> >> I grabbed the cvs snapshot gentoo's been using, but I don't see any
> >> way to feed in a device list text file so I can create the
> >> /dev/console node root filesystems need without needing root access on
> >> the host to mknod.
> >>
> >> genext2fs has the -D option, which takes:
> >>
> >>   /dev d 755 0 0 - - - - -
> >>   /dev/console c 640 0 0 5 1 0 0 -
> >>
> >> gen_init_cpio takes:
> >>
> >>   dir /dev 755 0 0
> >>   nod /dev/console 660 0 0 c 5 1
> >>
> >> Is there a squashfs option I'm missing...?
> >
> > I'll delay the release of the 4.0 tools and see if I can
> > implement this option in the next day or two.
>
> The Squashfs 4.0 tools have been released on sourceforge
> (http://www.squashfs.org), and mksquashfs now includes support for
> fake devices and directories.
>
>  From README-4.0

Some suggested wordsmithing on the text you posted:

---

Mksquashfs now supports pseudo-files, which may be specified via the -p 
and -pf command line options.

Pseudo-files allow fake directories, character devices, and block devices to 
be added to the Squashfs filesystem being built without requiring them to be 
present in the source filesystem.  This allows a non-root user to create a 
squashfs filesystem containing device nodes and directories (but files) with 
arbitrary ownership/permissions.

The -p option specifies an individual pseudo-file, and takes a single argument 
such as -p "/dev d 755 root root".  The -p option may be repeated as 
necessary.

The -pf option specifies a file containing a list of pseudo-file definitions, 
one per line.

Pseudo-file definitions look like this:

  /dev d 755 root root
  /dev/chr_dev c 666 root root 100 1
  /dev/blk_dev b 444 0 0 200 200

The first field is the complete path to the new entry in the squashfs 
filesystem. (This path cannot currently contain any whitespace space 
characters.)  The second field is a single letter indicating the entry type.  
Currently supported types are d (directory), b (block device), and c (char 
device).  The third, fourth, and fifth fields are (respectively) octal mode, 
user id, and group ID.  UID and GID can be specified either by decimal number 
or using a name valid on the host filesystem.

Directory pseudo-file definitions only contain five fields.  Device 
pseudo-file definitions include major and minor numbers in the sixth and 
seventh fields.

To ensure the pseudo-file definition is parsed as one argument, and to
ensure it isn't interpreted by the shell, the definition should be enclosed
in quotes when using the -p option, i.e.

  mksquashfs test test.sqsh -p "/dev/chrdev c 666 root 100 1"

If a directory given in a pseudo file path exists in the real source
directories it doesn't have to be specified as a pseudo directory, if it
doesn't exist in the source directories, it must first be specified as a 
pseudo directory (otherwise any pseudo devices under the directory will be
ignored).

For example

  /dev d 755 root root
  /dev/chr_dev c 666 root root 100 1

---

Thanks, looks good.  I should be able to test it out in the morning.

Rob
-- 
GPLv3 is to GPLv2 what Attack of the Clones is to The Empire Strikes Back.