Extend initramfs archive format to support xattrs

Extend initramfs archive format to support xattrs

[Mimi Zohar]

Hi James,

I think we need to describe the basic problem being discussed, make a
list of the problems and then decide which could and should be addressed
before upstreaming the initramfs xattr support.

The patches posted about a year ago to extend the initramfs archive
format to support xattrs changed the initramfs magic number from 070701
to 070703.  James pointed out, in an offline email, a number of problems
that should be addressed, before making this magic number change.

James' original list:
1) Bad CRC it just added bytes together not even a real crc much less
something that would cause confidence like md5 or shaX, or better a
digital signature. of course we are not even using that CRC yet.
2) It is still missing the other two timestamps all three of which
should likely be extended beyond 64bits if you want sub-second accuracy
3) The missing user and group names
4) Lack of padding/blocking control; having the file data uncompress on
a page boundary would be very convenient for the tmpfs
5) Handling sparse files
6) Alternate streams, for example Mac OS and NTFS
7) The longer device major and minor numbers.
 
Do we need to address all of these issues?  Are there any other changes,
which should be made before changing the magic number?

Mimi

Re: Extend initramfs archive format to support xattrs

[Rob Landley]

On 12/14/2015 07:17 AM, Mimi Zohar wrote:
> Hi James,
> 
> I think we need to describe the basic problem being discussed, make a
> list of the problems and then decide which could and should be addressed
> before upstreaming the initramfs xattr support.

I still have this on my todo list for toybox and would very much like to
be involved.

Any archive format change should add xattrs, fix the 32 bit time with no
nanoseconds problem too, and probably the 32 bit file size limit while
we're there.

Beyond that, cpio was chosen because it's simple. Let's not complicate
it unnecessarily?

> The patches posted about a year ago to extend the initramfs archive
> format to support xattrs changed the initramfs magic number from 070701
> to 070703.  James pointed out, in an offline email, a number of problems
> that should be addressed, before making this magic number change.

Here's my suggestion:

For the new type just change all the fields from 32 bits to 64 bits
(I.E. from 8 to 16 hex digits), and tack on xattrcount. With data
compression the difference more or less vanishes, and the header size
still fits in 256 bytes.

This keeps the parsing simple (all fields after the first are the same
size!) while solving not only filesize and timestamp (interpreted now as
seconds=mtime>>20; nanosecs=mtime&((1<<20)-1); which gives us half a
million years and change), but this also addresses the 32 bit inode and
nlink limits (in case anyone cares).

If xattrcount is nonzero, then at the end of this entry's datastream
have a 16 digit size, followed by that much data, repeated xattrcount
times. Each one is a key=value and the point of size is we don't care
what the value is (embedded NUL? No problem!).

If somebody wants a "data fork" instead of an xattr, pick a magic key
name for it. Not our problem. Smack vs Selinux: not our problem.

> James' original list:
> 1) Bad CRC it just added bytes together not even a real crc much less
> something that would cause confidence like md5 or shaX, or better a
> digital signature. of course we are not even using that CRC yet.

The enclosing compression checksums the data, no need to repeat it. As
you say, we're not currently using the cpio checksum. We're using the
gzip (or lzma...) checksum.

Heck, using the checksum field _as_ the xattrcount field in the new
format would work for me, but I don't feel strongly about the issue. If
you instead want to take 64 bits of data and use... I dunno, sha1sum
with the high and low halves xored, up to you.

> 2) It is still missing the other two timestamps all three of which
> should likely be extended beyond 64bits if you want sub-second accuracy

Why? Initramfs doesn't particularly need to distinguish creation,
modification, and access time.

We should extend beyond 32 bits for y2038 reasons. (Yeah unsigned gives
us almost another century but as long as we're changing the format
anyway...) A 64 bit format with 44 bits of time and 20 bits of
nanoseconds seems straightforward enough and gets us half a million years.

Again, not strongly opposed, just don't see a use case for it. Tar
exists if you want fancy.

> 3) The missing user and group names

We have existing uid and gid names. Are you suggesting that the kernel's
initramfs parser should reach into userspace, read the /etc/passwd file
that initramfs supplied, and confirm that those uid and gid names _match_?

I'm not following this suggestion at all. The kernel adding magic
uid/gid values for devtmpfs was deeply questionable, but at least it was
still numbers not names.

> 4) Lack of padding/blocking control; having the file data uncompress on
> a page boundary would be very convenient for the tmpfs

I don't see how that's cpio's problem? The data currently gets copied
into the page cache to align it: life is good.

The alternative you're suggesting seems to be putting each file's cpio
header and a lot of padding on its own page, which means you eat one
page per file during decompression, which can get big fast _and_ means
you've made confetti of physical memory in early boot when you discard
them all.

If you want to make the cpio extractor cleverer you can have it
decompress 64k at a time and handle the decompression in cache-friendly
chunks. Filling up and processing an output buffer is a thing they have
to be able to do. The processing is usually just write() in userspace
but it's a hook we can use. Then process just the data in that buffer
before extracting the rest (special case if you end halfway through a
cpio header copy it to the start of the buffer and do a decompress of
64k-remainder, but that's not a big deal).

The advantage of that is your extracted cpio copy should fit into L2
cache even on low-end chips, and be able to flush a page at a time to
DRAM in nice friendly sequential bursts to the same bank. (Heck, 32k
should be plenty and that's cache local for almost everything, modulo
the compressor's internal data probably not being so for anything but
gzip...)

This does not require any change to the file format.

> 5) Handling sparse files

Detect whole-page runs of zeroes during decompression. Gzip the file and
they basically go away. This does not require a format change, and
adding one is unnecessary complexity.

> 6) Alternate streams, for example Mac OS and NTFS

We're proposing adding xattr support. They contain arbitrary data and
are of arbitrary length. Pick a magic xattr name for your stream, not
our problem.

The above "xattr length, xattr contents" means you can have any in-band
data you want, we don't mess with the contents other than expecting a
"name=" header identifying each one.

> 7) The longer device major and minor numbers.

Currently devmajor is 32 bits and devminor is 32 bits. You need longer
than that?

Ok, this proposal would extended them to 64 bits anyway (treat all the
fields the same and simplify the parsing as an array), but I don't
understand the objection...

> Do we need to address all of these issues?

No?

> Are there any other changes,
> which should be made before changing the magic number?

My cpio.txt isn't on this machine, but not off the top of my head? File
size, file date, and xattr are the big ones.

> Mimi

Rob