Bug report: kernel hangs when mounting a crafted xfs image

Bug report: kernel hangs when mounting a crafted xfs image

[Xu, Wen]

When mounting a crafted xfs v4 image, the kernel hangs and never returns for mount operation. Suspect potential deadlock in log recovery exists. Not sure it is considered as a bug or not.

- Reproduce (on 4.17 upstream kernel)
# mkdir mnt
# mount -t xfs 0.img mnt

The image file (0.img.zip) is available here: https://bugzilla.kernel.org/attachment.cgi?id=276475

Thanks,
Wen

Re: Bug report: kernel hangs when mounting a crafted xfs image

[Xu, Wen]

I think the following kernel information should help to figure out the problem:

[ 1451.029259] INFO: task mount:1296 blocked for more than 120 seconds.
[ 1451.030575]       Not tainted 4.17.0+ #1
[ 1451.031356] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1451.032946] mount           D    0  1296   1286 0x00000000
[ 1451.032950] Call Trace:
[ 1451.032968]  __schedule+0x3ec/0x890
[ 1451.032972]  ? ___slab_alloc+0x2a9/0x540
[ 1451.032974]  schedule+0x36/0x80
[ 1451.032978]  xlog_grant_head_wait+0xb8/0x1e0
[ 1451.032980]  xlog_grant_head_check+0xf1/0x100
[ 1451.032982]  xfs_log_reserve+0xcb/0x1e0
[ 1451.032984]  xfs_trans_reserve+0x169/0x1d0
[ 1451.032986]  xfs_trans_alloc+0xbc/0x180
[ 1451.032989]  xlog_recover_process_intents.isra.42+0x189/0x270
[ 1451.032992]  xlog_recover_finish+0x21/0xa0
[ 1451.032994]  ? xlog_recover_finish+0x21/0xa0
[ 1451.032996]  xfs_log_mount_finish+0x64/0xe0
[ 1451.032999]  xfs_mountfs+0x5df/0x930
[ 1451.033002]  xfs_fs_fill_super+0x487/0x650
[ 1451.033005]  mount_bdev+0x17b/0x1b0
[ 1451.033007]  ? xfs_test_remount_options+0x60/0x60
[ 1451.033009]  xfs_fs_mount+0x15/0x20
[ 1451.033011]  mount_fs+0x3d/0x150
[ 1451.033014]  ? __alloc_percpu+0x15/0x20
[ 1451.033018]  vfs_kern_mount+0x67/0x110
[ 1451.033020]  do_mount+0x201/0xd00
[ 1451.033023]  ? memdup_user+0x42/0x60
[ 1451.033025]  ksys_mount+0x83/0xd0
[ 1451.033027]  __x64_sys_mount+0x25/0x30
[ 1451.033030]  do_syscall_64+0x5a/0x110
[ 1451.033033]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1451.033036] RIP: 0033:0x7fbae025cb9a
[ 1451.033036] Code: Bad RIP value.
[ 1451.033045] RSP: 002b:00007ffdcea92278 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
[ 1451.033057] RAX: ffffffffffffffda RBX: 0000000001747030 RCX: 00007fbae025cb9a
[ 1451.033060] RDX: 0000000001747210 RSI: 0000000001748f30 RDI: 000000000174fec0
[ 1451.033064] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000012
[ 1451.033068] R10: 00000000c0ed0000 R11: 0000000000000206 R12: 000000000174fec0
[ 1451.033072] R13: 0000000001747210 R14: 0000000000000000 R15: 0000000000000003

Thanks,
Wen

> On Jun 11, 2018, at 3:36 PM, Xu, Wen <wen.xu@gatech.edu> wrote:
> 
> When mounting a crafted xfs v4 image, the kernel hangs and never returns for mount operation. Suspect potential deadlock in log recovery exists. Not sure it is considered as a bug or not.
> 
> - Reproduce (on 4.17 upstream kernel)
> # mkdir mnt
> # mount -t xfs 0.img mnt
> 
> The image file (0.img.zip) is available here: https://bugzilla.kernel.org/attachment.cgi?id=276475
> 
> Thanks,
> Wen

Re: Bug report: kernel hangs when mounting a crafted xfs image

[Dave Chinner]

On Mon, Jun 11, 2018 at 07:36:48PM +0000, Xu, Wen wrote:
> When mounting a crafted xfs v4 image, the kernel hangs and never returns for mount operation. Suspect potential deadlock in log recovery exists. Not sure it is considered as a bug or not.
> 
> - Reproduce (on 4.17 upstream kernel)
> # mkdir mnt
> # mount -t xfs 0.img mnt
> 
> The image file (0.img.zip) is available here: https://bugzilla.kernel.org/attachment.cgi?id=276475

So it's a filesystem with a log that has bit corruptions in it's log
record headers, and log CRCs are turned off. All kernels since late
2012 have CRC'd log records for both v4 and v5 filesystems.  Hence
any remotely recent kernel faced wih bit errors in the journal are
going to behave differently - corruption warnings will be issued
first, and while v4 filesystems will continue to try to mount, a v5
filesystem will refuse to mount. And kernels old enough not to CRC
log records are unlikely to ever get bug fixes for maliciously
corrupted log records.

As such, I'm going to consider this a low priority right now.
Protecting log recovery against anything more than random bit errors
is a fairly major undertaking, and not something I plan on doing
in the near term...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Bug report: kernel hangs when mounting a crafted xfs image

[Xu, Wen]

Hi Dave,

I feel I can get your point a bit. Anyway recently I am fuzzing log section of XFS.
I reported four new bugs to mail list today. 
They are all memory corruption issues, and I think some of them are
enough to be used to perform controlled write and code execution. And I feel these bugs are not complex ones
but exist there that cannot simply be covered by CRC protection? In my humble opinions, CRC protection
mainly hardens the integrity of the metadata instead of eliminating malicious metadata leading to unexpected 
behaviors. I am not very sure considering my limited knowledge on XFS but you can take a look on them and
see the severity of these issues. 

Thanks,
Wen

> On Jun 12, 2018, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
> 
> On Mon, Jun 11, 2018 at 07:36:48PM +0000, Xu, Wen wrote:
>> When mounting a crafted xfs v4 image, the kernel hangs and never returns for mount operation. Suspect potential deadlock in log recovery exists. Not sure it is considered as a bug or not.
>> 
>> - Reproduce (on 4.17 upstream kernel)
>> # mkdir mnt
>> # mount -t xfs 0.img mnt
>> 
>> The image file (0.img.zip) is available here: https://bugzilla.kernel.org/attachment.cgi?id=276475
> 
> So it's a filesystem with a log that has bit corruptions in it's log
> record headers, and log CRCs are turned off. All kernels since late
> 2012 have CRC'd log records for both v4 and v5 filesystems.  Hence
> any remotely recent kernel faced wih bit errors in the journal are
> going to behave differently - corruption warnings will be issued
> first, and while v4 filesystems will continue to try to mount, a v5
> filesystem will refuse to mount. And kernels old enough not to CRC
> log records are unlikely to ever get bug fixes for maliciously
> corrupted log records.
> 
> As such, I'm going to consider this a low priority right now.
> Protecting log recovery against anything more than random bit errors
> is a fairly major undertaking, and not something I plan on doing
> in the near term...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: Bug report: kernel hangs when mounting a crafted xfs image

[Dave Chinner]

On Tue, Jun 12, 2018 at 11:30:45PM +0000, Xu, Wen wrote:
> Hi Dave,
> 
> I feel I can get your point a bit. Anyway recently I am fuzzing
> log section of XFS.  I reported four new bugs to mail list today.
> They are all memory corruption issues, and I think some of them
> are enough to be used to perform controlled write and code
> execution. And I feel these bugs are not complex ones but exist
> there that cannot simply be covered by CRC protection?

If CRCs are enabled, then they fail when there are random bit
corruptions in the journal. That stops these *storage errors* from
causing further problems. Fuzzing filesystems was originally
introduced to modelling storage errors and determine where the
filesystem error handling was deficient.

However, since "fuzzing" was taken over by the security crowd yeas
ago, it's all about malicious actors. I don't know how many times I
have to say "we cannot defend against malicious actors" before that
sinks in.

Again: CRCs are intended to mitigate the threat model of storage
level errors causing problems, not to defeat /malicious attackers/.
Modelling malicous users by random bit perturbation without CRC
protection exposes fundamental architectural trust model
deficiencies we cannot fix, and playing whack-a-mole with /malicious
fuzzing/ does not change that. Existing filesystems cannot win that
game, and it does nothing to protect users against emerging storage
error models and trends. I care about data storage, not malicious
actors.

> In my
> humble opinions, CRC protection mainly hardens the integrity of
> the metadata instead of eliminating malicious metadata leading to
> unexpected behaviors.

Key word in that statement: /malicious/

> I am not very sure considering my limited
> knowledge on XFS but you can take a look on them and see the
> severity of these issues. 

They are input checking problems, and log recovery does not have
robust input checking. That's why it has CRC protection - so we can
detect storage based errors to avoid parsing broken input.

We really need to improve the log recovery input verification, but
it's way more complex than any other subsystem we currently do
verification on. Untangling that mess will take time, resources and
careful planning. It's not a fast fix, but we'll get there. In the
mean time, throwing large numbers of "malicious actor" fuzzer bugs at
us doesn't help anyone.

If you want to do really useful fuzzing, try finding problems that
the v5 filesystem CRC checking doesn't detect and protect against.
That's something directly useful to us and hence XFS users, because
it indicates structures on disk we haven't fully protected from
typical storage based bit corruptions.

Put simply: I don't really care about bugs from "malicious image
corruption". I care about bugs that arise from storage errors that
we don't already catch, and that means v5 filesystems need to be the
focus of filesystem fuzzing....

I *really* don't want to have to say this again.

-Dave.
-- 
Dave Chinner
david@fromorbit.com