EXT4-fs (device loop0): panic forced after error -- bug or not?

EXT4-fs (device loop0): panic forced after error -- bug or not?

[Vegard Nossum]

[-- Attachment #1: Type: text/plain, Size: 2983 bytes --]

Hi,

Using the attached fuzzed ext4 image, I get the following panic on
latest linus/master.

I'm a bit unsure about whether this is really a bug or not; it looks
like the filesystem is set to panic on error but I still find it weird
that this behaviour is allowed by default (would it still panic if
somebody inserted this filesystem on a USB stick and it got automounted?).

I call mount() with mountflags=0 and data=NULL followed by opendir() and
readdir(), but if I just a manual mount + ls from the shell I don't see
the panic at all, just some of the errors, so I thought maybe there's
some sort of race somewhere?

Anyway, here's the log:

[EXT4 FS bs=1024, gc=2, bpg=8192, ipg=2048, mo=a802ec48, mo2=0002]
System zones: 1-2, 66-67, 82-83, 98-609, 8193-8194
EXT4-fs (loop0): mounted filesystem with writeback data mode. Opts: (null)
EXT4-fs warning (device loop0): dx_probe:782: inode #2: comm mount.exe: 
dx entry: limit 46 != root limit 125
EXT4-fs warning (device loop0): dx_probe:853: inode #2: comm mount.exe: 
Corrupt directory, running e2fsck is recommended
EXT4-fs error (device loop0): ext4_readdir:224: inode #2: block 68: comm 
mount.exe: path /mnt/ext4: bad entry in directory: rec_len is smaller 
than minimal - offset=0(0), inode=0, rec_len=0, name_len=0
Aborting journal on device loop0-8.
Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error

CPU: 1 PID: 971 Comm: mount.exe Not tainted 4.4.0-rc3+ #244
  ffff88001580adf0 ffff880015df7960 ffffffff81610cc9 ffffffff820a4540
  ffff880015df7a28 ffffffff811aaf62 0000000041b58ab3 ffffffff824627f4
  ffffffff811aae32 ffff880015df79d0 ffffffff00000010 ffff880015df7a38
Call Trace:
  [<ffffffff81610cc9>] dump_stack+0x44/0x5b
  [<ffffffff811aaf62>] panic+0x130/0x279
  [<ffffffff811aae32>] ? set_ti_thread_flag+0xf/0xf
  [<ffffffff8132f2ec>] ext4_handle_error.part.196+0x11c/0x120
  [<ffffffff8132f8a1>] __ext4_error_file+0x181/0x300
  [<ffffffff8132f720>] ? __ext4_error_inode+0x2b0/0x2b0
  [<ffffffff8127dc0d>] ? __find_get_block+0x13d/0x170
  [<ffffffff81282066>] ? __getblk_gfp+0x26/0x60
  [<ffffffff812f95e4>] ? ext4_getblk+0x104/0x270
  [<ffffffff812f94e0>] ? ext4_get_block_write_nolock+0x10/0x10
  [<ffffffff81617c7d>] ? radix_tree_lookup+0xd/0x10
  [<ffffffff812ecf90>] __ext4_check_dir_entry+0x150/0x220
  [<ffffffff812ed66c>] ext4_readdir+0x60c/0x1150
  [<ffffffff812ed060>] ? __ext4_check_dir_entry+0x220/0x220
  [<ffffffff8157c067>] ? selinux_file_permission+0x177/0x1d0
  [<ffffffff8124ac24>] iterate_dir+0x104/0x1e0
  [<ffffffff8124aed6>] SyS_getdents+0xe6/0x160
  [<ffffffff8124adf0>] ? SyS_old_readdir+0xf0/0xf0
  [<ffffffff8124a710>] ? SyS_ioctl+0x80/0x80
  [<ffffffff8106fc25>] ? trace_do_page_fault+0x45/0xf0
  [<ffffffff81f7d82e>] entry_SYSCALL_64_fastpath+0x12/0x71

Beware that the filesystem image changes after mounting and the
resulting filesystem doesn't panic again AFAICT, so it's better to make
a fresh copy before mounting.

I can test patches. Thanks,


Vegard

[-- Attachment #2: ext4.0.bz2 --]
[-- Type: application/x-bzip, Size: 1427 bytes --]

Re: EXT4-fs (device loop0): panic forced after error -- bug or not?

[Theodore Ts'o]

On Tue, Dec 01, 2015 at 03:22:02PM +0100, Vegard Nossum wrote:
> 
> I'm a bit unsure about whether this is really a bug or not; it looks
> like the filesystem is set to panic on error but I still find it weird
> that this behaviour is allowed by default (would it still panic if
> somebody inserted this filesystem on a USB stick and it got automounted?).

If the distribution cares about this, it should automount with mount
option "errors=remount-ro".

> I call mount() with mountflags=0 and data=NULL followed by opendir() and
> readdir(), but if I just a manual mount + ls from the shell I don't see
> the panic at all, just some of the errors, so I thought maybe there's
> some sort of race somewhere?

I'm not able to reproduce this; I'm getting EACCES to the opendir().

But looking at your kernel messages, it's not a bug.

Cheers,

						- Ted

Re: EXT4-fs (device loop0): panic forced after error -- bug or not?

[Vegard Nossum]

On 12/01/2015 04:32 PM, Theodore Ts'o wrote:
> On Tue, Dec 01, 2015 at 03:22:02PM +0100, Vegard Nossum wrote:
>>
>> I'm a bit unsure about whether this is really a bug or not; it looks
>> like the filesystem is set to panic on error but I still find it weird
>> that this behaviour is allowed by default (would it still panic if
>> somebody inserted this filesystem on a USB stick and it got automounted?).
>
> If the distribution cares about this, it should automount with mount
> option "errors=remount-ro".
>
>> I call mount() with mountflags=0 and data=NULL followed by opendir() and
>> readdir(), but if I just a manual mount + ls from the shell I don't see
>> the panic at all, just some of the errors, so I thought maybe there's
>> some sort of race somewhere?
>
> I'm not able to reproduce this; I'm getting EACCES to the opendir().
>
> But looking at your kernel messages, it's not a bug.

Alright, I'll add errors=remount-ro to my scripts.

Strange that you get EACCESS, without errors=remount-ro I get:

open("/dev/loop0", O_RDWR)              = 3
mkdir("/mnt/ext4", 0755)                = -1 EEXIST (File exists)
open("ext4.0", O_RDWR)                  = 4
ioctl(3, LOOP_SET_FD, 0x4)              = 0
close(4)                                = 0
ioctl(3, LOOP_SET_STATUS64, {offset=0, number=0, flags=0, 
file_name="ext4.0", ...}) = 0
mount("/dev/loop0", "/mnt/ext4", "ext4", 0, NULL) = 0
open("/mnt/ext4", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, <panic>

If I do add errors=remount-ro, I get:

open("/dev/loop0", O_RDWR)              = 3
mkdir("/mnt/ext4", 0755)                = -1 EEXIST (File exists)
open("ext4.0", O_RDWR)                  = 4
ioctl(3, LOOP_SET_FD, 0x4)              = 0
close(4)                                = 0
ioctl(3, LOOP_SET_STATUS64, {offset=0, number=0, flags=0, 
file_name="ext4.0", ...}) = 0
mount("/dev/loop0", "/mnt/ext4", "ext4", 0, "errors=remount-ro") = 0
open("/mnt/ext4", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0

Thanks for the quick response.


Vegard

Re: EXT4-fs (device loop0): panic forced after error -- bug or not?

[Theodore Ts'o]

On Tue, Dec 01, 2015 at 05:59:07PM +0100, Vegard Nossum wrote:
> Strange that you get EACCESS, without errors=remount-ro I get:
> 
> open("/dev/loop0", O_RDWR)              = 3
> mkdir("/mnt/ext4", 0755)                = -1 EEXIST (File exists)
> open("ext4.0", O_RDWR)                  = 4
> ioctl(3, LOOP_SET_FD, 0x4)              = 0
> close(4)                                = 0
> ioctl(3, LOOP_SET_STATUS64, {offset=0, number=0, flags=0,
> file_name="ext4.0", ...}) = 0
> mount("/dev/loop0", "/mnt/ext4", "ext4", 0, NULL) = 0
> open("/mnt/ext4", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
> getdents(4, <panic>

What is the source code of your test program, and what version of C
library are you using?  On my system, opendir() is getting translated
to:

openat(AT_FDCWD, "/vdc", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)

(It would have saved me time if you had sent me the source of your
test program, BTW.)

Cheers,

					- Ted

Re: EXT4-fs (device loop0): panic forced after error -- bug or not?

[Vegard Nossum]

On 12/02/2015 02:41 PM, Theodore Ts'o wrote:
> On Tue, Dec 01, 2015 at 05:59:07PM +0100, Vegard Nossum wrote:
>> Strange that you get EACCESS, without errors=remount-ro I get:
>>
>> open("/dev/loop0", O_RDWR)              = 3
>> mkdir("/mnt/ext4", 0755)                = -1 EEXIST (File exists)
>> open("ext4.0", O_RDWR)                  = 4
>> ioctl(3, LOOP_SET_FD, 0x4)              = 0
>> close(4)                                = 0
>> ioctl(3, LOOP_SET_STATUS64, {offset=0, number=0, flags=0,
>> file_name="ext4.0", ...}) = 0
>> mount("/dev/loop0", "/mnt/ext4", "ext4", 0, NULL) = 0
>> open("/mnt/ext4", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
>> getdents(4, <panic>
>
> What is the source code of your test program, and what version of C
> library are you using?  On my system, opendir() is getting translated
> to:
>
> openat(AT_FDCWD, "/vdc", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)
>
> (It would have saved me time if you had sent me the source of your
> test program, BTW.)

Hi Ted,

Sorry for dropping the ball on you here.

My test program is simply doing the loopback setup + mount() +
readdir(opendir(<mountpoint>)), there's nothing special about it.
Unfortunately, company policy prohibits me from sharing the actual code.

Compiling a simple readdir(opendir(".")) yields the same result:

open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
brk(0)                                  = 0xa87000
brk(0xab0000)                           = 0xab0000
getdents(3, /* 29 entries */, 32768)    = 904

This particular install is glibc 2.13-38+deb7u8, so it's admittedly
fairly old.

However, if you say it's not a bug when not mounted with
errors=remount-ro, I don't expect anybody to keep looking into this.


Vegard