e2fsck doesn't finish

e2fsck doesn't finish

[Julius Schwartzenberg]

Hi all,

I'm running into an e2fsck issue which seems to occur with one
particular filesystem. The version from Ubuntu, Fedora and the latest
release compiled from source all exhibit the same issue.

On one filesystem I have, when I run e2fsck, it prints it's doing step 1
and then after a while just:
"e2fsck: aborted"
Nothing else is printed. I tried a badblock scan on the partition, but
that didn't reveal any badblocks.

Here is the output of dumpe2fs:
https://schwart6.home.xs4all.nl/dumpe2fs.sda4.gz

I suspect there's some condition with that filesystem that e2fsck
doesn't handle. What would be the best way to figure out more about this
problem?

Best regards,
Julius

Re: e2fsck doesn't finish

[Eric Sandeen]

On 4/16/17 12:07 PM, Julius Schwartzenberg wrote:
> Hi all,
> 
> I'm running into an e2fsck issue which seems to occur with one
> particular filesystem. The version from Ubuntu, Fedora and the latest
> release compiled from source all exhibit the same issue.
> 
> On one filesystem I have, when I run e2fsck, it prints it's doing step 1
> and then after a while just:
> "e2fsck: aborted"
> Nothing else is printed. I tried a badblock scan on the partition, but
> that didn't reveal any badblocks.
> 
> Here is the output of dumpe2fs:
> https://schwart6.home.xs4all.nl/dumpe2fs.sda4.gz
> 
> I suspect there's some condition with that filesystem that e2fsck
> doesn't handle. What would be the best way to figure out more about this
> problem?

Providing an e2image might allow developers to analyze the problem.
If it's a large filesystem, the qcow output format is most portable.

-Eric

Re: e2fsck doesn't finish

[Julius Schwartzenberg]

Eric Sandeen wrote:
> Providing an e2image might allow developers to analyze the problem.
> If it's a large filesystem, the qcow output format is most portable.

Here is the e2image:
https://schwart6.home.xs4all.nl/sda4.e2i.bz2

The file is around 150M.
It's made with -s as the file system concerned is actually /home on my
notebook.

Thanks,
Julius

Re: e2fsck doesn't finish

[Eric Sandeen]

On 4/17/17 10:11 AM, Julius Schwartzenberg wrote:
> Eric Sandeen wrote:
>> Providing an e2image might allow developers to analyze the problem.
>> If it's a large filesystem, the qcow output format is most portable.
> 
> Here is the e2image:
> https://schwart6.home.xs4all.nl/sda4.e2i.bz2
> 
> The file is around 150M.
> It's made with -s as the file system concerned is actually /home on my
> notebook.

Got it, you may wish to delete it now just to err on the side of
privacy.

Thanks,
-Eric

Re: e2fsck doesn't finish

[Eric Sandeen]

On 4/16/17 12:07 PM, Julius Schwartzenberg wrote:
> Hi all,
> 
> I'm running into an e2fsck issue which seems to occur with one
> particular filesystem. The version from Ubuntu, Fedora and the latest
> release compiled from source all exhibit the same issue.
> 
> On one filesystem I have, when I run e2fsck, it prints it's doing step 1
> and then after a while just:
> "e2fsck: aborted"
> Nothing else is printed. I tried a badblock scan on the partition, but
> that didn't reveal any badblocks.
> 
> Here is the output of dumpe2fs:
> https://schwart6.home.xs4all.nl/dumpe2fs.sda4.gz
> 
> I suspect there's some condition with that filesystem that e2fsck
> doesn't handle. What would be the best way to figure out more about this
> problem?

When I look at your image, it's finding tons of corruption - much
more than just "step 1" followed by "e2fsck: aborted"

If you didn't see that, then I wonder if e2image is not properly
handling your filesystem...

You might try running with the '-v' flag and see if it gives any
more hints about where it's at when it fails, and attach the
full e2fsck output to the next reply.

-Eric

Re: e2fsck doesn't finish

[Theodore Ts'o]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

On Mon, Apr 17, 2017 at 02:16:26PM -0500, Eric Sandeen wrote:
> 
> When I look at your image, it's finding tons of corruption - much
> more than just "step 1" followed by "e2fsck: aborted"

I also downloaded the image, and what I saw was a large number of
messages of the form:

HTREE directory inode 4195634 has an invalid root node.
Clear HTree index? yes

I'm prety sure this is a result of the e2image -S (scramble) option.

Followed by "e2fsck: aborted"

It looks like the problem is fixed with the latest version of
e2fsprogs, though.  I just tried running using a build of e2fsprogs
from the next branch, and this seems to allow e2fsck to fully fix the
file system and avoid the "aborted" message.  The interesting thing is
even after it is fixed, there is something which is causing the 1.43.4
version of e2fsprogs abort:

e2fsck 1.43.5-WIP (17-Feb-2017)
Pass 1: Checking inodes, blocks, and sizes
Timestamp(s) on inode 4200169 beyond 2310-04-04 are likely pre-1970.
Fix? yes

Timestamp(s) on inode 4201357 beyond 2310-04-04 are likely pre-1970.
Fix? yes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

home: ***** FILE SYSTEM WAS MODIFIED *****
home: 2185285/16179200 files (9.7% non-contiguous), 58130079/64713836 blocks
<tytso@callcc> {/home/tytso}
1012% /sbin/e2fsck -fy /tmp/test.img
e2fsck 1.43.4 (31-Jan-2017)
Pass 1: Checking inodes, blocks, and sizes
e2fsck: aborted

The obvious first guess is that there was a bug in the date encoding
validity checking.


     	      	      	       	     	      - Ted

[-- Attachment #2: sda4.log.gz --]
[-- Type: application/gzip, Size: 9620 bytes --]

Re: e2fsck doesn't finish

[Julius Schwartzenberg]

Theodore Ts'o wrote:
> It looks like the problem is fixed with the latest version of
> e2fsprogs, though.  I just tried running using a build of e2fsprogs
> from the next branch, and this seems to allow e2fsck to fully fix the
> file system and avoid the "aborted" message.

I just tried the latest e2fsck from git and it indeed fully got through.
I'm really glad here and I hope there will be a new release of e2fsprogs
soon!

I still have a (dd) copy of the original full image on my external hard
drive in case it would be useful.

I don't know if this is relevant, but maybe it could also be related to
a conversion? Till recently the filesystem was still ext3 and I was only
using it from Ubuntu 16.04. After installing Fedora 25 on another
partition, Ubuntu couldn't mount this partition anymore and I had to
change its entry in /etc/fstab from ext3 to ext4. Then it could mount
it, but Ubuntu's e2fsck complained there were features in use that it
didn't support. I reported this Fedora bug then:
https://bugzilla.redhat.com/show_bug.cgi?id=1440429

After I reported bug, I ran into the issues with e2fsck failing to complete.

In any case, thanks a lot Theodore and Eric for looking into this! My
issue is solved now, but if there's anything that would still be useful
to figure out I'll be glad to provide any information that could help.

Thanks!
Julius

Re: e2fsck doesn't finish

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 2063 bytes --]

On Apr 19, 2017, at 2:57 PM, Julius Schwartzenberg <julius.schwartzenberg@gmail.com> wrote:
> 
> Theodore Ts'o wrote:
>> It looks like the problem is fixed with the latest version of
>> e2fsprogs, though.  I just tried running using a build of e2fsprogs
>> from the next branch, and this seems to allow e2fsck to fully fix the
>> file system and avoid the "aborted" message.
> 
> I just tried the latest e2fsck from git and it indeed fully got through.
> I'm really glad here and I hope there will be a new release of e2fsprogs
> soon!
> 
> I still have a (dd) copy of the original full image on my external hard
> drive in case it would be useful.
> 
> I don't know if this is relevant, but maybe it could also be related to
> a conversion? Till recently the filesystem was still ext3 and I was only
> using it from Ubuntu 16.04. After installing Fedora 25 on another
> partition, Ubuntu couldn't mount this partition anymore and I had to
> change its entry in /etc/fstab from ext3 to ext4. Then it could mount
> it, but Ubuntu's e2fsck complained there were features in use that it
> didn't support. I reported this Fedora bug then:
> https://bugzilla.redhat.com/show_bug.cgi?id=1440429

It would be useful to know which feature(s) that Fedora 25 enabled that
Ubuntu 16.04 didn't understand.  You could use something like:

    dumpe2fs -h /dev/XXX | grep -i feature

under Ubuntu.  Typically, the code shouldn't be enabling any feature on
the filesystem without an explicit request from the admin.

Also, the "failed to mount" case should print something to the console
logs about an unsupported feature, which would help isolate the problem.

Cheers, Andreas

> After I reported bug, I ran into the issues with e2fsck failing to complete.
> 
> In any case, thanks a lot Theodore and Eric for looking into this! My
> issue is solved now, but if there's anything that would still be useful
> to figure out I'll be glad to provide any information that could help.
> 
> Thanks!
> Julius


Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: e2fsck doesn't finish

[Julius Schwartzenberg]

Andreas Dilger wrote:
> It would be useful to know which feature(s) that Fedora 25 enabled that
> Ubuntu 16.04 didn't understand.  You could use something like:
> 
>     dumpe2fs -h /dev/XXX | grep -i feature
> 
> under Ubuntu.  Typically, the code shouldn't be enabling any feature on
> the filesystem without an explicit request from the admin.

I expect that Ubuntu's kernel (4.4.0) supports more features than its
e2fsck as I can mount it now. Ubuntu 16.04 comes with e2fsprogs version
1.42.13. The feature that that version says it doesn't support is
inline_data.


> Also, the "failed to mount" case should print something to the console
> logs about an unsupported feature, which would help isolate the problem.

Yes, it printed it couldn't mount the filesystem as ext3 originally, but
I changed /etc/fstab accordingly.

I have always been quite ignorant on filesystem upgrades and features
supported by either the kernel or e2fsprogs and upgrades/downgrades of
existing filesystems as usually everything just works.
I expect that the upgrade from ext3 to ext4 was triggered by Fedora's
installer (not e2fsprogs) and that dual boot configurations are less
tested in general.

Best regards,
Julius