Q. cache in squashfs?

["J. R. Okajima" <hooanon05@yahoo.co.jp>]

Hello Phillip,

I've found an intersting issue about squashfs.
Please give me a guidance or an advise.
In short:
Why does squashfs read and decompress the same block several times?
Is the nested fs-image always better for squashfs?

Long:
I created two squashfs images.
- from /bin directly by mksquashfs
  $ mksquashfs /bin /tmp/a.img
- from a single ext3 fs image which contains /bin
  $ dd if=/dev/zero of=/tmp/ext3/img bs=... count=...
  $ mkfs -t ext3 -F -m 0 -T small -O dir_index /tmp/ext3/img
  $ sudo mount -o loop /tmp/ext3/img /mnt
  $ tar -C /bin -cf - . | tar -C /mnt -xpf -
  $ sudo umount /mnt
  $ mksquashfs /tmp/ext3/img /tmp/b.img

Of course, /tmp/b.img is bigger than /tmp/a.img. It is OK.
For these squashfs, I tried profiling the random file read all over the
fs.
$ find /squashfs -type f > /tmp/l
$ seq 10 | time sh -c "while read i; do rl /tmp/l | xargs -r cat & done > /dev/null; wait"
("rl" is a command to randomize lines)

For b.img, I have to loopback-mount twice.
$ mount -o ro,loop /tmp/b.img /tmp/sq
$ mount -o ro,loop /tmp/sq/img /mnt

Honestly speaking, I gueesed b.img is slower due to the nested fs
overhead. But it shows that b.img (ext3 within squashfs) consumes less
CPU cycles and faster.

- a.img (plain squashfs)
  0.00user 0.14system 0:00.09elapsed 151%CPU (0avgtext+0avgdata 2192maxresident)k
  0inputs+0outputs (0major+6184minor)pagefaults 0swaps

(oprofile report)
samples  %        image name               app name                 symbol name
710      53.9514  zlib_inflate.ko          zlib_inflate             inflate_fast
123       9.3465  libc-2.7.so              libc-2.7.so              (no symbols)
119       9.0426  zlib_inflate.ko          zlib_inflate             zlib_adler32
106       8.0547  zlib_inflate.ko          zlib_inflate             zlib_inflate
95        7.2188  ld-2.7.so                ld-2.7.so                (no symbols)
64        4.8632  oprofiled                oprofiled                (no symbols)
36        2.7356  dash                     dash                     (no symbols)

- b.img (ext3 + squashfs)
  0.00user 0.01system 0:00.06elapsed 22%CPU (0avgtext+0avgdata 2192maxresident)k
  0inputs+0outputs (0major+6134minor)pagefaults 0swaps

samples  %        image name               app name                 symbol name
268      37.0678  zlib_inflate.ko          zlib_inflate             inflate_fast
126      17.4274  libc-2.7.so              libc-2.7.so              (no symbols)
106      14.6611  ld-2.7.so                ld-2.7.so                (no symbols)
57        7.8838  zlib_inflate.ko          zlib_inflate             zlib_adler32
45        6.2241  oprofiled                oprofiled                (no symbols)
40        5.5325  dash                     dash                     (no symbols)
33        4.5643  zlib_inflate.ko          zlib_inflate             zlib_inflate


The biggest difference is to decompress the blocks.
(Since /bin is used for this sample, the difference is not so big. But
if I used antoher dir which has much more files than /bin, then the
difference grows too).
I don't think the difference of fs-layout or metadata is a problem.
Actually inserting debug-prints to show the block index in
squashfs_read_data(), it shows squashfs reads the same block multiple
times from a.img.

int squashfs_read_data(struct super_block *sb, void **buffer, u64 index,
			int length, u64 *next_index, int srclength, int pages)
{
	:::
	// for datablock
	for (b = 0; bytes < length; b++, cur_index++) {
		bh[b] = sb_getblk(sb, cur_index);
+		pr_info("%llu\n", cur_index);
		if (bh[b] == NULL)
			goto block_release;
		bytes += msblk->devblksize;
	}
	ll_rw_block(READ, b, bh);
	:::
	// for metadata
	for (; bytes < length; b++) {
		bh[b] = sb_getblk(sb, ++cur_index);
+		pr_info("%llu\n", cur_index);
		if (bh[b] == NULL)
			goto block_release;
		bytes += msblk->devblksize;
	}
	ll_rw_block(READ, b - 1, bh + 1);
	:::
}

In case of b.img, the same block is read several times too. But the
number of times is much smaller than a.img.

I am intrested where did the difference come from.
Do you think the loopback block device in the middle cached the
decompressed block effectively?
- a.img
  squashfs
  + loop0
  + disk

- b.img
  ext3
  + loop1	<-- so effective?
  + squashfs
  + loop0
  + disk

In other word, is inserting a loopback mount always effective for all
squashfs?


Thanx for reading long mail
J. R. Okajima