silo trouble with ext3

silo trouble with ext3

[Meelis Roos]

Short summary: silo can not find any files from my ext3. Was this 
the problem recently discussed here about silo?

Debian unstable's silo 1.4.14+git20120127-1 and 110G ext3 filesystem. 
The same machine worked fine for years, running all the kernels and 
Debian unstable. Last successful reboot was less than a week ago, and 
Debian was updated after that (probably updating silo package).
Today I compiled 3.3-rc4+git and rebooted to try it, and silo can not 
find anything. silo itself loads but can not open neither /etc/silo.conf 
nor any kernel image by direct name:

Rebooting with command: boot                                          
Boot device: disk  File and args: 
SILO Version 1.4.14

Cannot find /etc/silo.conf (error: 4294967295)

Couldn't load /etc/silo.conf
No config file loaded, you can boot just from this command line
Type [prompath;]part/path_to_image [parameters] on the prompt
E.g. /iommu/sbus/espdma/esp/sd@3,0;4/vmlinux root=/dev/sda4
or 2/vmlinux.live (to load vmlinux.live from 2nd partition of boot disk)
boot: 1/boot/test
Allocated 64 Megs of memory at 0x40000000 for kernel

Cannot find /boot/test (error: 4294967295)

Image not found.... try again
boot:


Tried fsck.ext3 -f, did not help. Re-installed silo from netboot 
chroot, did not help either.

This is Ultra 5 with OBP 3.31 (latest available), so should not have any 
1G or 2G limits on boot partition?

Ideas?

-- 
Meelis Roos (mroos@linux.ee)

Re: silo trouble with ext3

[crn]

I had a similar problem a long time ago, I created a small boot partition
at the beginning of the disk, ext2. The problem went away but I did not
figure out why.
ISTR there were issues with large disks at various times, for example the
Ultra 5/10 was only sold with 8Gb drives and there were issues at 32Gb.
But ICBW.


On Tue, 21 Feb 2012, Meelis Roos wrote:

> Short summary: silo can not find any files from my ext3. Was this
> the problem recently discussed here about silo?
>
> Debian unstable's silo 1.4.14+git20120127-1 and 110G ext3 filesystem.
> The same machine worked fine for years, running all the kernels and
> Debian unstable. Last successful reboot was less than a week ago, and
> Debian was updated after that (probably updating silo package).
> Today I compiled 3.3-rc4+git and rebooted to try it, and silo can not
> find anything. silo itself loads but can not open neither /etc/silo.conf
> nor any kernel image by direct name:
>
> Rebooting with command: boot
> Boot device: disk  File and args:
> SILO Version 1.4.14
>
> Cannot find /etc/silo.conf (error: 4294967295)
>
> Couldn't load /etc/silo.conf
> No config file loaded, you can boot just from this command line
> Type [prompath;]part/path_to_image [parameters] on the prompt
> E.g. /iommu/sbus/espdma/esp/sd@3,0;4/vmlinux root=/dev/sda4
> or 2/vmlinux.live (to load vmlinux.live from 2nd partition of boot disk)
> boot: 1/boot/test
> Allocated 64 Megs of memory at 0x40000000 for kernel
>
> Cannot find /boot/test (error: 4294967295)
>
> Image not found.... try again
> boot:
>
>
> Tried fsck.ext3 -f, did not help. Re-installed silo from netboot
> chroot, did not help either.
>
> This is Ultra 5 with OBP 3.31 (latest available), so should not have any
> 1G or 2G limits on boot partition?
>
> Ideas?
>
> -- 
> Meelis Roos (mroos@linux.ee)
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: silo trouble with ext3

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Tue, 21 Feb 2012 15:43:07 +0200 (EET)

> Short summary: silo can not find any files from my ext3. Was this 
> the problem recently discussed here about silo?
> 
> Debian unstable's silo 1.4.14+git20120127-1 and 110G ext3 filesystem. 
> The same machine worked fine for years, running all the kernels and 
> Debian unstable. Last successful reboot was less than a week ago, and 
> Debian was updated after that (probably updating silo package).
> Today I compiled 3.3-rc4+git and rebooted to try it, and silo can not 
> find anything. silo itself loads but can not open neither /etc/silo.conf 
> nor any kernel image by direct name:

I really wish they hadn't pushed this new SILO into debian without a
couple months more testing of the new ext{2,3,4} code I wrote.

I knew there would be regressions like this and it's not ready at all
for mass consumption.

Re: silo trouble with ext3

[David Miller]

From: crn@netunix.com
Date: Tue, 21 Feb 2012 18:03:12 +0000 (GMT)

> 
> I had a similar problem a long time ago, I created a small boot
> partition at the beginning of the disk, ext2. The problem went away
> but I did not figure out why.

This is totally unrelated.

Meelis's problem is due to the complete rewrite of the ext{2,3,4}
support in SILO which I committed recently, and which propagated into
Debian way too quickly.

Re: silo trouble with ext3

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Tue, 21 Feb 2012 15:43:07 +0200 (EET)

> Image not found.... try again
> boot:

Ok, let's try to fix this.  Can you run "dumpe2fs -h $DEV" where $DEV
is the block device the filesystem resides on, and report the output?

Using that I'll try to create a partition that resembles your's in
hopes I can reproduce the bug.

Thanks.

Re: silo trouble with ext3

[Jurij Smakov]

On Tue, Feb 21, 2012 at 02:36:40PM -0500, David Miller wrote:
> From: Meelis Roos <mroos@linux.ee>
> Date: Tue, 21 Feb 2012 15:43:07 +0200 (EET)
> 
> > Short summary: silo can not find any files from my ext3. Was this 
> > the problem recently discussed here about silo?
> > 
> > Debian unstable's silo 1.4.14+git20120127-1 and 110G ext3 filesystem. 
> > The same machine worked fine for years, running all the kernels and 
> > Debian unstable. Last successful reboot was less than a week ago, and 
> > Debian was updated after that (probably updating silo package).
> > Today I compiled 3.3-rc4+git and rebooted to try it, and silo can not 
> > find anything. silo itself loads but can not open neither /etc/silo.conf 
> > nor any kernel image by direct name:
> 
> I really wish they hadn't pushed this new SILO into debian without a
> couple months more testing of the new ext{2,3,4} code I wrote.

That would be my handywork :-).

> I knew there would be regressions like this and it's not ready at all
> for mass consumption.

To clarify, the new SILO has only appeared in Debian testing (wheezy) 
and unstable (sid) distributions and not in the released version 
(squeeze), to which it will never propagate. I've announced new SILO 
availability for early testing a few days before uploading it to 
unstable [0] and then it spent the usual 10 days in unstable before 
propagating to testing [1] (any bug reported against the new version 
during that time would have blocked this propagation). 

If needed, the version can be rolled back pretty easily in 
testing/unstable (will take just a couple of days), but that will 
minimize the chances of the new version getting any mileage. Given 
that the next Debian release is something like 6 months to a year away 
and the fixes will be forthcoming sooner than that :-), I don't really 
see a reason to do such a rollback, especially considering that vast 
majority of the Debian sparc users, who followed the default 
installation procedure, will not be affected by it. Default install 
still creates a small ext2 partition at the beginning of the disk for 
kernels and initrds (legacy of the sparc32 days, when kernel would 
refuse to boot if it was further than 1GB away from the beginning of 
the disk, or something like that). That was the reason why I did not 
get any problem reports and have not seen any problems on my machine. 

Meelis, sorry for the trouble. You can restore your machine's ability 
to boot by installing the stable SILO version [2]. Please let me know 
off-list if you would like more detailed recovery instructions.

[0] http://lists.debian.org/debian-sparc/2012/01/msg00038.html
[1] http://packages.qa.debian.org/s/silo.html
[2] http://ftp.ie.debian.org/debian/pool/main/s/silo/silo_1.4.14+git20100228-1+b1_sparc.deb

Best regards,
-- 
Jurij Smakov                                           jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                      KeyID: C99E03CC

Re: silo trouble with ext3

[David Miller]

From: Jurij Smakov <jurij@wooyd.org>
Date: Tue, 21 Feb 2012 22:51:08 +0000

> If needed, the version can be rolled back pretty easily in 
> testing/unstable (will take just a couple of days), but that will 
> minimize the chances of the new version getting any mileage. Given 
> that the next Debian release is something like 6 months to a year away 
> and the fixes will be forthcoming sooner than that :-), I don't really 
> see a reason to do such a rollback, especially considering that vast 
> majority of the Debian sparc users, who followed the default 
> installation procedure, will not be affected by it. Default install 
> still creates a small ext2 partition at the beginning of the disk for 
> kernels and initrds (legacy of the sparc32 days, when kernel would 
> refuse to boot if it was further than 1GB away from the beginning of 
> the disk, or something like that). That was the reason why I did not 
> get any problem reports and have not seen any problems on my machine. 

The assumption here is that it has something to do with ext3 or some
ext3 specific feature, but it might not.

For example, I just found a bug with symlink handling in path
traversal fixed by the following patch.  So if there are symlinks
involved in Meelis's setup that would be the cause rather than not
specifically using ext2 for the boot partition.

--------------------
ext2: Fix symlink being overwritten.

When the symlink length is small, it's contained in the block list
part of the inode.  But we still need to copy the symlink into the
caller's buffer in that case because this string is used for path
traversal which will read an inode and thus overwrite the inode the
symlink string is inside of.

Also, forcefully NULL terminate the symlink string.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 second/fs/ext2.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/second/fs/ext2.c b/second/fs/ext2.c
index 710695a..53f2b5e 100644
--- a/second/fs/ext2.c
+++ b/second/fs/ext2.c
@@ -450,9 +450,11 @@ static char *read_symlink(struct silo_ext2_state *s, struct ext2_inode *ip, char
 	__u32 isize = ext2_to_cpu_32(ip->i_size);
 
 	if (isize <= 60)
-		return (char *) &ip->i_block[0];
+		strncpy(namebuf, (char *) &ip->i_block[0], isize);
+	else
+		read_data(s, ext2_to_cpu_32(ip->i_block[0]), 0, isize, namebuf);
 
-	read_data(s, ext2_to_cpu_32(ip->i_block[0]), 0, isize, namebuf);
+	namebuf[isize] = '\0';
 
 	return namebuf;
 }
-- 
1.7.7.6

Re: silo trouble with ext3

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 21 Feb 2012 18:10:07 -0500 (EST)

> For example, I just found a bug with symlink handling in path
> traversal fixed by the following patch.  So if there are symlinks
> involved in Meelis's setup that would be the cause rather than not
> specifically using ext2 for the boot partition.
> 
> --------------------
> ext2: Fix symlink being overwritten.

Ignore me, there is no bug.  All callers specifically use a special
on-stack inode buffer to avoid this problem.

Re: silo trouble with ext3

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 21 Feb 2012 18:33:10 -0500 (EST)

> From: David Miller <davem@davemloft.net>
> Date: Tue, 21 Feb 2012 18:10:07 -0500 (EST)
> 
>> For example, I just found a bug with symlink handling in path
>> traversal fixed by the following patch.  So if there are symlinks
>> involved in Meelis's setup that would be the cause rather than not
>> specifically using ext2 for the boot partition.
>> 
>> --------------------
>> ext2: Fix symlink being overwritten.
> 
> Ignore me, there is no bug.  All callers specifically use a special
> on-stack inode buffer to avoid this problem.

Ok, while waiting for the debug dump from Meelis I think I found the
problem.

With my test harness I reproduced a case similar to what Meelis saw
but it has nothing to do with ext3'ness or anything like that.

The block group descriptors were having their location miscalculated.
It only worked for the first block of group descriptors.

So if an inode is outside of the first several block groups, things
don't work.  I guess for most /boot partitions, which are relatively
small, most if not all of the files fit into the working range.

Jurij, can you possibly build a test package for Meelis to see if this
fixes the reported bug?

It's pushed out to the silo GIT repo as well if that helps any.

--------------------
ext2: Calculate group descriptor location properly.

Calculate the block and offset correctly for group descriptors.

The existing code would work properly only for the first block
of descriptors.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 second/fs/ext2.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/second/fs/ext2.c b/second/fs/ext2.c
index 710695a..7eb665f 100644
--- a/second/fs/ext2.c
+++ b/second/fs/ext2.c
@@ -57,6 +57,8 @@ struct silo_ext2_state {
 	__u32			block_size;
 	__u32			addr_per_block;
 	__u32			addr_per_block_bits;
+	__u32			desc_per_block;
+	__u32			desc_per_block_bits;
 };
 static struct silo_ext2_state *sstate;
 
@@ -285,8 +287,8 @@ static void read_group(struct silo_ext2_state *s, __u32 grp_no,
 	__u32 first = ext2_to_cpu_32(s->super->s_first_data_block);
 	__u32 blk, offset;
 
-	blk = first + 1;
-	offset = grp_no * sizeof(*grp);
+	blk = first + 1 + (grp_no >> s->desc_per_block_bits);
+	offset = (grp_no & (s->desc_per_block - 1)) * sizeof(*grp);
 
 	read_data(s, blk, offset, sizeof(*grp), grp);
 }
@@ -371,6 +373,8 @@ static int open_ext2(char *device)
 
 	s->addr_per_block = s->block_size / sizeof(__u32);
 	s->addr_per_block_bits = calc_ilog2(s->addr_per_block);
+	s->desc_per_block = s->block_size / sizeof(struct ext2_group_desc);
+	s->desc_per_block_bits = calc_ilog2(s->desc_per_block);
 
 	s->scratch_block[0] = malloc(s->block_size * 4);
 	if (!s->scratch_block[0]) {
-- 
1.7.7.6

Re: silo trouble with ext3

[Meelis Roos]

> Ok, let's try to fix this.  Can you run "dumpe2fs -h $DEV" where $DEV
> is the block device the filesystem resides on, and report the output?

I see you already have a patch for testing but here is the output from 
mounted filesystem:

Filesystem volume name:   mandariin-root
Last mounted on:          <not available>
Filesystem UUID:          1ff8d6d7-6d17-4bfa-9758-27a879320627
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype 
sparse_super
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         not clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              14446080
Block count:              28835840
Reserved block count:     1441792
Free blocks:              7187096
Free inodes:              14323802
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16416
Inode blocks per group:   513
Filesystem created:       Thu Sep 23 17:14:35 2004
Last mount time:          Tue Feb 21 12:24:06 2012
Last write time:          Wed Feb 22 09:20:16 2012
Mount count:              1
Maximum mount count:      27
Last checked:             Tue Feb 21 13:32:08 2012
Check interval:           15552000 (6 months)
Next check after:         Sun Aug 19 14:32:08 2012
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      4605a528-34b9-4b7c-8455-fdcaf274e145
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             32M
Journal length:           8192
Journal sequence:         0x00917abb
Journal start:            0


-- 
Meelis Roos (mroos@linux.ee)

Re: silo trouble with ext3

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Wed, 22 Feb 2012 09:22:51 +0200 (EET)

>> Ok, let's try to fix this.  Can you run "dumpe2fs -h $DEV" where $DEV
>> is the block device the filesystem resides on, and report the output?
> 
> I see you already have a patch for testing but here is the output from 
> mounted filesystem:

That's a pretty large partition for /boot or do you put everything on
one partition, including / and /boot together?

Re: silo trouble with ext3

[Meelis Roos]

> >> Ok, let's try to fix this.  Can you run "dumpe2fs -h $DEV" where $DEV
> >> is the block device the filesystem resides on, and report the output?
> > 
> > I see you already have a patch for testing but here is the output from 
> > mounted filesystem:
> 
> That's a pretty large partition for /boot or do you put everything on
> one partition, including / and /boot together?

Yes, all files on one partition as a test - on some machines I have 
/home separate, apparently not there. Small /boot becomes full too 
easily (like when bisecting something) and when I need to do manual 
partion setup, I usually just use / without separate /boot. I have 
understood that this should work with ext2 and ext3 in silo. Not with 
ext4 yet - does silo start to fully grok ext4 when you latest e2fslibs 
changes integration is complete?

Should I try grub2 on some machines too? My museum sparcs are all like a 
big testbed, most do not run anything important and I can do test 
installes too when I happen to have some time.

-- 
Meelis Roos (mroos@linux.ee)

Re: silo trouble with ext3

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Wed, 22 Feb 2012 09:32:54 +0200 (EET)

> Yes, all files on one partition as a test - on some machines I have 
> /home separate, apparently not there. Small /boot becomes full too 
> easily (like when bisecting something) and when I need to do manual 
> partion setup, I usually just use / without separate /boot. I have 
> understood that this should work with ext2 and ext3 in silo. Not with 
> ext4 yet - does silo start to fully grok ext4 when you latest e2fslibs 
> changes integration is complete?

At the very least a basic ext4 partition with extents ought to
work but is untested by me.  The good news is that I have a special
userland framework I can link the ext2.c file into and use to test
SILO ext2 support without having to reboot or anything like that.

So all I need to do is make some interesting ext{2,3,4} filesystems
and run the userland tester against them.

This reminds me that I need to add some basic code to check the
feature flags in the ext{2,3,4} superblock and make sure we don't try
to mount an FS that uses something we don't support.  In particular I
think sparse superblocks will trip up my group descriptor
calculations.

> Should I try grub2 on some machines too? My museum sparcs are all like a 
> big testbed, most do not run anything important and I can do test 
> installes too when I happen to have some time.

I'd stay away from grub2 for the time being, I haven't tried it myself in
a very long time.  I was thinking of getting back to it today but someone
installed only 24 hours in a day.

Re: silo trouble with ext3

[Jurij Smakov]

On Tue, Feb 21, 2012 at 08:27:33PM -0500, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 21 Feb 2012 18:33:10 -0500 (EST)
> 
> > From: David Miller <davem@davemloft.net>
> > Date: Tue, 21 Feb 2012 18:10:07 -0500 (EST)
> > 
> >> For example, I just found a bug with symlink handling in path
> >> traversal fixed by the following patch.  So if there are symlinks
> >> involved in Meelis's setup that would be the cause rather than not
> >> specifically using ext2 for the boot partition.
> >> 
> >> --------------------
> >> ext2: Fix symlink being overwritten.
> > 
> > Ignore me, there is no bug.  All callers specifically use a special
> > on-stack inode buffer to avoid this problem.
> 
> Ok, while waiting for the debug dump from Meelis I think I found the
> problem.
> 
> With my test harness I reproduced a case similar to what Meelis saw
> but it has nothing to do with ext3'ness or anything like that.
> 
> The block group descriptors were having their location miscalculated.
> It only worked for the first block of group descriptors.
> 
> So if an inode is outside of the first several block groups, things
> don't work.  I guess for most /boot partitions, which are relatively
> small, most if not all of the files fit into the working range.
> 
> Jurij, can you possibly build a test package for Meelis to see if this
> fixes the reported bug?

Sure, a new package with this patch included is available at
http://www.wooyd.org/silo/

Best regards,
-- 
Jurij Smakov                                           jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                      KeyID: C99E03CC

Re: silo trouble with ext3

[mroos]

> > Jurij, can you possibly build a test package for Meelis to see if this
> > fixes the reported bug?
> 
> Sure, a new package with this patch included is available at
> http://www.wooyd.org/silo/

This silo works for me, thank you!

-- 
Meelis Roos (mroos@linux.ee)

Re: silo trouble with ext3

[David Miller]

From: mroos@linux.ee
Date: Thu, 23 Feb 2012 08:05:05 +0200 (EET)

>> > Jurij, can you possibly build a test package for Meelis to see if this
>> > fixes the reported bug?
>> 
>> Sure, a new package with this patch included is available at
>> http://www.wooyd.org/silo/
> 
> This silo works for me, thank you!

Thanks a lot for testing.