Squashfs mainlining status?

Squashfs mainlining status?

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 636 bytes --]

	Hi Phillip,

I was wondering what's the status of mainlining squashfs?
Is there anything I can do to help?

Thanks!

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: Squashfs mainlining status?

[Phillip Lougher]

Geert Uytterhoeven wrote:
> 	Hi Phillip,
> 
> I was wondering what's the status of mainlining squashfs?
> Is there anything I can do to help?
> 

I am currently working on this.  A re-write and merge of the Squashfs 
metadata and fragment cache code has been finished (it is in the 
Squashfs CVS tree).  The original implementation had a number of 
performance issues which I wanted to address before resubmission.

The major work of course is the move to a little endian layout without 
any bit fields in the structures to make it easier to swap member fields 
on big-endian architectures.  I'm currently working on this on a non-CVS 
branch.  Most of the work till now has been without any attempt to 
retain backwards compatibility with the current 3.3 little-endian 
filesystem layout.   After your private email regarding retaining 
backwards compatibility I will look again at this issue, it may well be 
that I will abandon my current layout changes that I have so far made.

I have taken time off work to work on the necessary mainline changes 
full time (unpaid unfortunately, so I need to look for another job 
soon).  Hopefully this will mean I will make better progress than I have 
been able to in the last couple of years.

Phillip

Re: [Squashfs-devel] Squashfs mainlining status?

[Peter Korsgaard]

>>>>> "Phillip" == Phillip Lougher <phillip@lougher.demon.co.uk> writes:

Hi,

 Phillip> I am currently working on this.  A re-write and merge of the
 Phillip> Squashfs metadata and fragment cache code has been finished
 Phillip> (it is in the Squashfs CVS tree).  The original
 Phillip> implementation had a number of performance issues which I
 Phillip> wanted to address before resubmission.

Nice to hear! Are you aiming for 2.6.27 or later?

 Phillip> The major work of course is the move to a little endian
 Phillip> layout without any bit fields in the structures to make it
 Phillip> easier to swap member fields on big-endian architectures.
 Phillip> I'm currently working on this on a non-CVS branch.
 Phillip> Most of the work till now has been without any attempt to
 Phillip> retain backwards compatibility with the current 3.3
 Phillip> little-endian filesystem layout.  After your private email
 Phillip> regarding retaining backwards compatibility I will look
 Phillip> again at this issue, it may well be that I will abandon my
 Phillip> current layout changes that I have so far made.

What private mail? Anything of interest to the rest of the group? As I
said earlier, I'm strongly in faviour of cleaning up the layout now
and get rid of all the backwards compatibility code. Doing it once
it's in mainline will be really hard.

 Phillip> I have taken time off work to work on the necessary mainline
 Phillip> changes full time (unpaid unfortunately, so I need to look
 Phillip> for another job soon).  Hopefully this will mean I will make
 Phillip> better progress than I have been able to in the last couple
 Phillip> of years.

;) Good luck with it. Is there anything I can do to help?

-- 
Bye, Peter Korsgaard

Re: [Squashfs-devel] Squashfs mainlining status?

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2371 bytes --]

	Hi Peter,

On Wed, 4 Jun 2008, Peter Korsgaard wrote:
> >>>>> "Phillip" == Phillip Lougher <phillip@lougher.demon.co.uk> writes:
>  Phillip> The major work of course is the move to a little endian
>  Phillip> layout without any bit fields in the structures to make it
>  Phillip> easier to swap member fields on big-endian architectures.
>  Phillip> I'm currently working on this on a non-CVS branch.
>  Phillip> Most of the work till now has been without any attempt to
>  Phillip> retain backwards compatibility with the current 3.3
>  Phillip> little-endian filesystem layout.  After your private email
>  Phillip> regarding retaining backwards compatibility I will look
>  Phillip> again at this issue, it may well be that I will abandon my
>  Phillip> current layout changes that I have so far made.
> 
> What private mail? Anything of interest to the rest of the group? As I

Oops, I've just returned from holidays, and realize I never actually responded
to this question. So here is the gist of it:

| I had a closer look at the structures and bitfields, and I still think it's
| possible to stay compatible with the current little endian layout:
|   1. About half of the bitfields are actually bytes or 16-bit words. Hence
|      these can easily be converted to __u8 or __le16.
|   2. Most of the remaining bitfields sit in an 8, 16, or 32 bit integer.
|      Hence these can be converted to such an integer (or an array of bytes,
|      cfr. 3 below), and accessed using get_unaligned_le{16,32}() and a
|      shift/mask operation.
|   3. There are a few bitfields that sit in an `odd' number of bytes. These
|      can be converted to an array of bytes, and accessed using
|      get_unaligned_le{16,32}() and a shift/mask operation.
|   4. The integral fields in the structs should be accessed using
|      get_unaligned_le{8,16,32}(), too.

I noticed the squashfs CVS repository is getting a new (incompatible) 4.0
layout now?

What do people think?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@eu.sony.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: [Squashfs-devel] Squashfs mainlining status?

[Robert Lougher]

On Thu, Jul 17, 2008 at 4:35 PM, Geert Uytterhoeven
<Geert.Uytterhoeven@sonycom.com> wrote:
>        Hi Peter,
>
> On Wed, 4 Jun 2008, Peter Korsgaard wrote:
>> >>>>> "Phillip" == Phillip Lougher <phillip@lougher.demon.co.uk> writes:
>>  Phillip> The major work of course is the move to a little endian
>>  Phillip> layout without any bit fields in the structures to make it
>>  Phillip> easier to swap member fields on big-endian architectures.
>>  Phillip> I'm currently working on this on a non-CVS branch.
>>  Phillip> Most of the work till now has been without any attempt to
>>  Phillip> retain backwards compatibility with the current 3.3
>>  Phillip> little-endian filesystem layout.  After your private email
>>  Phillip> regarding retaining backwards compatibility I will look
>>  Phillip> again at this issue, it may well be that I will abandon my
>>  Phillip> current layout changes that I have so far made.
>>
>> What private mail? Anything of interest to the rest of the group? As I
>
> Oops, I've just returned from holidays, and realize I never actually responded
> to this question. So here is the gist of it:
>
> | I had a closer look at the structures and bitfields, and I still think it's
> | possible to stay compatible with the current little endian layout:
> |   1. About half of the bitfields are actually bytes or 16-bit words. Hence
> |      these can easily be converted to __u8 or __le16.
> |   2. Most of the remaining bitfields sit in an 8, 16, or 32 bit integer.
> |      Hence these can be converted to such an integer (or an array of bytes,
> |      cfr. 3 below), and accessed using get_unaligned_le{16,32}() and a
> |      shift/mask operation.
> |   3. There are a few bitfields that sit in an `odd' number of bytes. These
> |      can be converted to an array of bytes, and accessed using
> |      get_unaligned_le{16,32}() and a shift/mask operation.
> |   4. The integral fields in the structs should be accessed using
> |      get_unaligned_le{8,16,32}(), too.
>
> I noticed the squashfs CVS repository is getting a new (incompatible) 4.0
> layout now?
>
> What do people think?
>

Last time I looked you were not the maintainer of squashfs.  Phillip
spent a long time examining the possibility of maintaining
compatibility but decided asthat there were sufficient advantages in a
new layout to warrant the change.

Rob.

> With kind regards,
>
> Geert Uytterhoeven
> Software Architect
>
> Sony Techsoft Centre Europe
> The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
>
> Phone:    +32 (0)2 700 8453
> Fax:      +32 (0)2 700 8622
> E-mail:   Geert.Uytterhoeven@eu.sony.com
> Internet: http://www.sony-europe.com/
>
> A division of Sony Europe (Belgium) N.V.
> VAT BE 0413.825.160 · RPR Brussels
> Fortis 293-0376800-10 GEBA-BE-BB
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Squashfs-devel mailing list
> Squashfs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/squashfs-devel
>
>

Re: [Squashfs-devel] Squashfs mainlining status?

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3304 bytes --]

	Hi Robert,

On Thu, 17 Jul 2008, Robert Lougher wrote:
> On Thu, Jul 17, 2008 at 4:35 PM, Geert Uytterhoeven
> <Geert.Uytterhoeven@sonycom.com> wrote:
> > On Wed, 4 Jun 2008, Peter Korsgaard wrote:
> >> >>>>> "Phillip" == Phillip Lougher <phillip@lougher.demon.co.uk> writes:
> >>  Phillip> The major work of course is the move to a little endian
> >>  Phillip> layout without any bit fields in the structures to make it
> >>  Phillip> easier to swap member fields on big-endian architectures.
> >>  Phillip> I'm currently working on this on a non-CVS branch.
> >>  Phillip> Most of the work till now has been without any attempt to
> >>  Phillip> retain backwards compatibility with the current 3.3
> >>  Phillip> little-endian filesystem layout.  After your private email
> >>  Phillip> regarding retaining backwards compatibility I will look
> >>  Phillip> again at this issue, it may well be that I will abandon my
> >>  Phillip> current layout changes that I have so far made.
> >>
> >> What private mail? Anything of interest to the rest of the group? As I
> >
> > Oops, I've just returned from holidays, and realize I never actually responded
> > to this question. So here is the gist of it:
> >
> > | I had a closer look at the structures and bitfields, and I still think it's
> > | possible to stay compatible with the current little endian layout:
> > |   1. About half of the bitfields are actually bytes or 16-bit words. Hence
> > |      these can easily be converted to __u8 or __le16.
> > |   2. Most of the remaining bitfields sit in an 8, 16, or 32 bit integer.
> > |      Hence these can be converted to such an integer (or an array of bytes,
> > |      cfr. 3 below), and accessed using get_unaligned_le{16,32}() and a
> > |      shift/mask operation.
> > |   3. There are a few bitfields that sit in an `odd' number of bytes. These
> > |      can be converted to an array of bytes, and accessed using
> > |      get_unaligned_le{16,32}() and a shift/mask operation.
> > |   4. The integral fields in the structs should be accessed using
> > |      get_unaligned_le{8,16,32}(), too.
> >
> > I noticed the squashfs CVS repository is getting a new (incompatible) 4.0
> > layout now?
> >
> > What do people think?
> 
> Last time I looked you were not the maintainer of squashfs.  Phillip

Sorry, I don't want to offend anyone. Nor do I intent to become the maintainer.
I just answered Peter's question, because I think people should know the
answer.

> spent a long time examining the possibility of maintaining
> compatibility but decided asthat there were sufficient advantages in a
> new layout to warrant the change.

If there are good reasons for breaking compatibility, we're more than happy to
accept that. Don't you think it would be a good idea to explain them to the
readers of the squashfs-devel mailing list?

Phillip, can you please elaborate? Many thanks in advance!

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@eu.sony.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: [Squashfs-devel] Squashfs mainlining status?

[Robert Lougher]

Hi,

On Fri, Jul 18, 2008 at 7:14 AM, Geert Uytterhoeven
<Geert.Uytterhoeven@sonycom.com> wrote:

> If there are good reasons for breaking compatibility, we're more than happy to
> accept that. Don't you think it would be a good idea to explain them to the
> readers of the squashfs-devel mailing list?
>
> Phillip, can you please elaborate? Many thanks in advance!
>

We need to wait for Phillip to reply for the details.  But my
understanding is that Phillip is using this as an opportunity to clean
up the layout and to remove limitations.  This may give Squashfs an
easier ride in mainlining reviews, and once Squashfs is mainlined
layout changes may become much more difficult, so it's wise to do it
now.

Rob.

> With kind regards,
>
> Geert Uytterhoeven
> Software Architect
>
> Sony Techsoft Centre Europe
> The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
>
> Phone:    +32 (0)2 700 8453
> Fax:      +32 (0)2 700 8622
> E-mail:   Geert.Uytterhoeven@eu.sony.com
> Internet: http://www.sony-europe.com/
>
> A division of Sony Europe (Belgium) N.V.
> VAT BE 0413.825.160 · RPR Brussels
> Fortis 293-0376800-10 GEBA-BE-BB