Re: [RFC][PATCH v2 0/5] Experiments with overlayfs filemap

Amir Goldstein
2019-01-25T11:24:28Z

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

> > ...Luckily, I have implemented ./run --ov --recycle way back, otherwise we
> > would have had very poor test coverage for persistency of overlayfs writes.
> >
> > So we currently have 6 xfstests failing on write persistency:
> > overlay/018 overlay/032 overlay/044 overlay/047 overlay/050 overlay/051
> >
> > and 2 unionmount tests (over /base xfs):
> > ./run --ov --samefs --recycle rename-new-pop-dir
> > ./run --ov --samefs --recycle rename-mass-dir
>
> Great progress. Thanks!
>
> I don't think you need to mess with dcache flushing, since this is all
> kernel (page cache) pages without any possible dcache aliasing issues
> that plague user pages. So a simple copy_page() should do.
>

Pardon my ignorance, but couldn't pages be mapped to user?
Anyway, it seems that generic_file_buffered_read() does
flush_dcache_page() if needed, so we need not worry about it.

> It would be even better to do it with O_DIRECT to remove cache
> duplication. Something like the attached...
>

Yes, definitely better. Thanks for the patch!

You'll be happy to know that the two unionmount tests fails on tmpfs
as expected (or maybe you already checked that).

I do get one new lockdep splat from xfstest, something we need to
look into (attached 013.dmesg).

One questions about your patch:
Skipping the ovl_change_flags() for upper realfile, that's just a
quick hack, right?
I'll need to fix this up properly.

About the way forward, implementing writepage() should be quite straight forward
from this, so I will add write support re-work the series and re-post.

I think I can now see how to implement the LOWER -> LOWER_SHARED ->
UPPER_SHARED transitions, but that will require some more juggling of
realfile flags etc.

With the current implementation of UPPER_SHARED only we can gain:
- No copy up latency on open() with metacopy=on
- Very improved ovl_sync_fs() (no longer sync all containers on syncfs(2))

Do you think we should aim for merging just UPPER_SHARED in dev
cycle to hash out the bugs of overlay page cache or aim for merging
LOWER_SHARED+UPPER_SHARED when it is ready?

Thanks,
Amir.

[-- Attachment #2: 013.dmesg --]
[-- Type: application/octet-stream, Size: 9576 bytes --]

[ 119.448268] run fstests overlay/013 at 2019-01-25 10:55:14
[ 119.755174] XFS (vde): Mounting V5 Filesystem
[ 119.774400] XFS (vde): Ending clean mount
[ 119.925944] XFS (vdf): Unmounting Filesystem
[ 119.981734] XFS (vdf): Mounting V5 Filesystem
[ 120.008546] XFS (vdf): Ending clean mount

[ 120.133970] ======================================================
[ 120.135229] WARNING: possible circular locking dependency detected
[ 120.136520] 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047 Not tainted
[ 120.137917] ------------------------------------------------------
[ 120.139182] test_upper/9289 is trying to acquire lock:
[ 120.140250] 00000000c03c6afd (&inode->i_rwsem){++++}, at: xfs_ilock+0x18b/0x208
[ 120.141745]
but task is already holding lock:
[ 120.142929] 000000002e44e1ff (&mm->mmap_sem){++++}, at: __do_page_fault+0x1b4/0x3ff
[ 120.144388]
which lock already depends on the new lock.

[ 120.146072]
the existing dependency chain (in reverse order) is:
[ 120.147652]
-> #1 (&mm->mmap_sem){++++}:
[ 120.148847] __might_fault+0x61/0x89
[ 120.149727] copy_page_to_iter+0x11f/0x2c1
[ 120.150474] generic_file_buffered_read+0x279/0x660
[ 120.151544] xfs_file_buffered_aio_read+0x117/0x135
[ 120.152557] xfs_file_read_iter+0x81/0xc8
[ 120.153538] __vfs_read+0x136/0x16e
[ 120.154359] vfs_read+0xb5/0x141
[ 120.155137] ksys_read+0x60/0xb1
[ 120.155920] do_syscall_64+0x57/0x14e
[ 120.156783] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 120.157920]
-> #0 (&inode->i_rwsem){++++}:
[ 120.159055] lock_acquire+0x141/0x16c
[ 120.159921] down_read_nested+0x45/0x9c
[ 120.160818] xfs_ilock+0x18b/0x208
[ 120.161522] xfs_file_dio_aio_read+0x110/0x145
[ 120.162489] xfs_file_read_iter+0x7a/0xc8
[ 120.163380] do_iter_readv_writev+0x14b/0x182
[ 120.164203] do_iter_read+0x87/0xfb
[ 120.164763] ovl_do_readpage+0x9e/0xed
[ 120.165341] ovl_readpage+0xf/0x22
[ 120.165915] filemap_fault+0x311/0x425
[ 120.166620] __do_fault+0x1d/0x67
[ 120.167343] __handle_mm_fault+0x823/0xae5
[ 120.168300] __do_page_fault+0x2fb/0x3ff
[ 120.169366] async_page_fault+0x1e/0x30
[ 120.170361] __clear_user+0x36/0x5f
[ 120.171305] padzero+0x1c/0x2b
[ 120.172034] load_elf_binary+0x795/0xdfe
[ 120.172870] search_binary_handler+0x72/0xdb
[ 120.173812] __do_execve_file.isra.11+0x708/0x9b2
[ 120.174837] do_execve+0x21/0x24
[ 120.175588] __x64_sys_execve+0x26/0x2b
[ 120.176451] do_syscall_64+0x57/0x14e
[ 120.177274] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 120.178364]
other info that might help us debug this:

[ 120.179929] Possible unsafe locking scenario:

[ 120.181086] CPU0 CPU1
[ 120.181974] ---- ----
[ 120.182866] lock(&mm->mmap_sem);
[ 120.183549] lock(&inode->i_rwsem);
[ 120.184735] lock(&mm->mmap_sem);
[ 120.185746] lock(&inode->i_rwsem);
[ 120.186235]
*** DEADLOCK ***

[ 120.187019] 1 lock held by test_upper/9289:
[ 120.187728] #0: 000000002e44e1ff (&mm->mmap_sem){++++}, at: __do_page_fault+0x1b4/0x3ff
[ 120.189372]
stack backtrace:
[ 120.190268] CPU: 0 PID: 9289 Comm: test_upper Not tainted 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047
[ 120.192277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 120.194139] Call Trace:
[ 120.194669] dump_stack+0x67/0x8e
[ 120.195539] print_circular_bug.isra.18+0x26e/0x27b
[ 120.196585] check_prev_add.constprop.26+0x12c/0x5f7
[ 120.197499] ? __lock_acquire+0xd80/0xee7
[ 120.198259] __lock_acquire+0xd80/0xee7
[ 120.199062] lock_acquire+0x141/0x16c
[ 120.199723] ? xfs_ilock+0x18b/0x208
[ 120.200352] ? xfs_file_dio_aio_read+0x110/0x145
[ 120.201158] down_read_nested+0x45/0x9c
[ 120.201829] ? xfs_ilock+0x18b/0x208
[ 120.202456] xfs_ilock+0x18b/0x208
[ 120.203057] xfs_file_dio_aio_read+0x110/0x145
[ 120.203933] xfs_file_read_iter+0x7a/0xc8
[ 120.204729] do_iter_readv_writev+0x14b/0x182
[ 120.205514] do_iter_read+0x87/0xfb
[ 120.206018] ovl_do_readpage+0x9e/0xed
[ 120.206676] ovl_readpage+0xf/0x22
[ 120.207177] ? ovl_write_begin+0xab/0xab
[ 120.208028] filemap_fault+0x311/0x425
[ 120.208802] ? ovl_fault+0xc9/0x16d
[ 120.209494] __do_fault+0x1d/0x67
[ 120.210152] __handle_mm_fault+0x823/0xae5
[ 120.210962] __do_page_fault+0x2fb/0x3ff
[ 120.211747] async_page_fault+0x1e/0x30
[ 120.212507] RIP: 0010:__clear_user+0x36/0x5f
[ 120.213202] Code: 71 20 37 82 53 48 89 f3 be 13 00 00 00 e8 a2 58 77 ff 0f 01 cb 48 89 d8 48 c1 eb 03 48 89 ef 83 e0 07 48 89 d9 48 85 c9 74 0f <48> c7 07 00 00 00 00 48 83 c7 08 ff c9 75 f1 48 89 c1 85 c9 74 0a
[ 120.217055] RSP: 0018:ffffc90002f37d88 EFLAGS: 00050206
[ 120.218119] RAX: 0000000000000000 RBX: 00000000000001f5 RCX: 00000000000001f5
[ 120.219217] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000563302b8b058
[ 120.220578] RBP: 0000563302b8b058 R08: 00000000df1fc5d3 R09: 0000000000000006
[ 120.221953] R10: 0000000000000000 R11: 0000000000000001 R12: ffff888079794900
[ 120.223331] R13: 0000000000000000 R14: 0000563302b8b070 R15: 0000563302b8b058
[ 120.224675] ? __clear_user+0x1e/0x5f
[ 120.225192] padzero+0x1c/0x2b
[ 120.225614] load_elf_binary+0x795/0xdfe
[ 120.226172] search_binary_handler+0x72/0xdb
[ 120.226828] __do_execve_file.isra.11+0x708/0x9b2
[ 120.227685] do_execve+0x21/0x24
[ 120.228452] __x64_sys_execve+0x26/0x2b
[ 120.229447] do_syscall_64+0x57/0x14e
[ 120.230161] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 120.231137] RIP: 0033:0x7fc14348a647
[ 120.231846] Code: Bad RIP value.
[ 120.232476] RSP: 002b:00007ffc28f4b638 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[ 120.233529] RAX: ffffffffffffffda RBX: 00000000013cf568 RCX: 00007fc14348a647
[ 120.234450] RDX: 000000000162f008 RSI: 00000000013d4f68 RDI: 00000000013cf568
[ 120.235382] RBP: 0000000000000000 R08: 0000000000710710 R09: 0000000000000000
[ 120.236840] R10: 000000000000059a R11: 0000000000000202 R12: 0000000000000000
[ 120.238271] R13: 00000000013d4f68 R14: 000000000162f008 R15: 0000000000000000
[ 120.246570] ------------[ cut here ]------------
[ 120.247604] downgrading a read lock
[ 120.247618] WARNING: CPU: 1 PID: 9289 at /home/amir/build/src/linux/kernel/locking/lockdep.c:3553 lock_downgrade+0xe3/0x17b
[ 120.250592] CPU: 1 PID: 9289 Comm: test_upper Not tainted 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047
[ 120.252718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 120.254537] RIP: 0010:lock_downgrade+0xe3/0x17b
[ 120.255537] Code: 00 00 00 8b 44 24 04 89 83 60 08 00 00 48 8b 45 00 48 89 83 58 08 00 00 f6 45 32 03 74 0e 48 c7 c7 79 7a 28 82 e8 77 d8 fa ff <0f> 0b 8a 45 32 4c 89 6d 08 44 89 e6 48 89 df 83 e0 fc 83 c8 01 88
[ 120.259903] RSP: 0018:ffffc90002f37e28 EFLAGS: 00010082
[ 120.261047] RAX: 0000000000000000 RBX: ffff88807a3e02c0 RCX: 0000000000000000
[ 120.262509] RDX: ffffffff810fb24f RSI: 0000000000000001 RDI: ffffffff810fb261
[ 120.263887] RBP: ffff88807a3e0b28 R08: 0000000000000017 R09: 0000000000000004
[ 120.265338] R10: 0000000000000000 R11: ffffc90002f37c47 R12: 0000000000000001
[ 120.266666] R13: ffffffff811e4bb3 R14: 0000000000000246 R15: ffff88805ca38168
[ 120.268122] FS: 00007f371215ebc0(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[ 120.269597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 120.270361] CR2: 00007f37115fd1b0 CR3: 000000007c470006 CR4: 00000000003606e0
[ 120.271299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 120.272776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 120.274323] Call Trace:
[ 120.274863] downgrade_write+0x17/0x81
[ 120.275698] __do_munmap+0x26f/0x2fe
[ 120.276437] __vm_munmap+0x74/0xbf
[ 120.277138] __x64_sys_munmap+0x27/0x2c
[ 120.277931] do_syscall_64+0x57/0x14e
[ 120.278697] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 120.279738] RIP: 0033:0x7f3711f5c447
[ 120.280475] Code: b4 eb 91 f7 d8 89 05 f8 bc 20 00 48 c7 c0 ff ff ff ff eb a1 f7 d8 89 05 e7 bc 20 00 e9 73 ff ff ff 66 90 b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8d 0d c9 bc 20 00 f7 d8 89 01 48 83
[ 120.284277] RSP: 002b:00007ffe35bb37c8 EFLAGS: 00000202 ORIG_RAX: 000000000000000b
[ 120.285603] RAX: ffffffffffffffda RBX: 0000004060eef67d RCX: 00007f3711f5c447
[ 120.286542] RDX: 0000004000000000 RSI: 0000000000002739 RDI: 00007f3712160000
[ 120.287592] RBP: 00007ffe35bb39e0 R08: 0000000000000000 R09: 00007f3711583b98
[ 120.288936] R10: 00007f3712167030 R11: 0000000000000202 R12: 0000000000000000
[ 120.289887] R13: 00007f3712168170 R14: 00007f371215ebc0 R15: 00007f3712168170
[ 120.290881] irq event stamp: 1547
[ 120.291374] hardirqs last enabled at (1547): [<ffffffff81a73a36>] _raw_spin_unlock_irq+0x29/0x33
[ 120.293412] hardirqs last disabled at (1546): [<ffffffff81a737ba>] _raw_spin_lock_irq+0x10/0x67
[ 120.295143] softirqs last enabled at (1182): [<ffffffff81e0035e>] __do_softirq+0x35e/0x39a
[ 120.296779] softirqs last disabled at (873): [<ffffffff810a7902>] irq_exit+0x62/0xb1
[ 120.298291] ---[ end trace c1aa87c93f028edf ]---
[ 120.320405] XFS (vde): Unmounting Filesystem