* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
From: Stephen Rothwell @ 2022-02-16 7:39 UTC (permalink / raw)
To: Sachin Sant; +Cc: linux-xfs, riteshh, linuxppc-dev, linux-next
In-Reply-To: <5AD0BD6A-2C31-450A-924E-A581CD454073@linux.ibm.com>
[-- Attachment #1: Type: text/plain, Size: 821 bytes --]
Hi Sachin,
On Wed, 16 Feb 2022 12:55:02 +0530 Sachin Sant <sachinp@linux.ibm.com> wrote:
>
> While running xfstests on IBM Power10 logical partition (LPAR) booted
> with 5.17.0-rc4-next-20220215 following warning was seen:
>
> The warning is seen when test tries to unmount the file system. This problem is seen
> while running generic/475 sub test. Have attached captured messages during the test
> run of generic/475.
>
> xfstest is a recent add to upstream regression bucket. I don’t have any previous data
> to attempt a git bisect.
If you have time, could you test v5.17-rc4-2-gd567f5db412e (the commit
in Linus' tree that next-20220215 is based on) and if that OK, then a
bisect from that to 5.17.0-rc4-next-20220215 may be helpful.
Thanks for the report.
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
From: Sachin Sant @ 2022-02-16 7:25 UTC (permalink / raw)
To: linux-xfs; +Cc: riteshh, linuxppc-dev, linux-next
[-- Attachment #1: Type: text/plain, Size: 6049 bytes --]
While running xfstests on IBM Power10 logical partition (LPAR) booted
with 5.17.0-rc4-next-20220215 following warning was seen:
[ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
[ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
[ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
[ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2547.389120] XFS (dm-0): log I/O error -5
[ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2547.662818] ------------[ cut here ]------------
[ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
[ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
[ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
[ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
[ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
[ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
[ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
[ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
[ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
[ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
[ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
[ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
[ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
[ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
[ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
[ 2547.662927] Call Trace:
[ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
[ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
[ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
[ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
[ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
[ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
[ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
[ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
[ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
[ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
[ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
[ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
[ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
[ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
[ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
[ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
[ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
[ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
[ 2547.663010] IRQMASK: 0
[ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
[ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
[ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
[ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
[ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
[ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
[ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
[ 2547.663047] --- interrupt: c00
[ 2547.663048] Instruction dump:
[ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
[ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
[ 2547.663062] ---[ end trace 0000000000000000 ]---
[ 2547.663075] XFS (dm-0): Unmounting Filesystem
The warning is seen when test tries to unmount the file system. This problem is seen
while running generic/475 sub test. Have attached captured messages during the test
run of generic/475.
xfstest is a recent add to upstream regression bucket. I don’t have any previous data
to attempt a git bisect.
Let me know if any additional information is required.
Thanks
-Sachin
[-- Attachment #2: xfstest.txt --]
[-- Type: text/plain, Size: 76413 bytes --]
[ 2523.250933] run fstests generic/475 at 2022-02-15 15:42:10
[ 2523.366239] XFS (sdb1): Mounting V5 Filesystem
[ 2523.376679] XFS (sdb1): Ending clean mount
[ 2523.378107] xfs filesystem being mounted at /mnt/test supports timestamps until 2038 (0x7fffffff)
[ 2523.502357] XFS (dm-0): Mounting V5 Filesystem
[ 2523.508526] XFS (dm-0): Ending clean mount
[ 2523.509735] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2524.516736] dm-0: writeback error on inode 33750928, offset 655360, sector 21228312
[ 2524.517884] dm-0: writeback error on inode 100730311, offset 589824, sector 63235832
[ 2524.518348] dm-0: writeback error on inode 67165541, offset 258371584, sector 42405248
[ 2524.518625] dm-0: writeback error on inode 136194, offset 417792, sector 212888
[ 2524.518727] dm-0: writeback error on inode 114612, offset 1052672, sector 295824
[ 2524.518876] buffer_io_error: 6 callbacks suppressed
[ 2524.518878] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2524.518888] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2524.518892] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2524.518895] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2524.518898] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2524.518901] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2524.518905] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2524.518908] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2524.518911] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2524.518914] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2524.519703] dm-0: writeback error on inode 67200889, offset 6684672, sector 42237480
[ 2524.520249] dm-0: writeback error on inode 100772183, offset 30445568, sector 63105160
[ 2524.520258] dm-0: writeback error on inode 100835769, offset 270336, sector 63079496
[ 2524.520266] dm-0: writeback error on inode 114527, offset 40960, sector 84968
[ 2524.520272] dm-0: writeback error on inode 67244195, offset 25001984, sector 42327408
[ 2524.520710] XFS (dm-0): log I/O error -5
[ 2524.520719] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2524.520780] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2524.793985] XFS (dm-0): Unmounting Filesystem
[ 2524.870119] XFS (dm-0): Mounting V5 Filesystem
[ 2525.015369] XFS (dm-0): Starting recovery (logdev: internal)
[ 2525.418197] XFS (dm-0): Ending recovery (logdev: internal)
[ 2525.419156] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2525.425641] XFS (dm-0): log I/O error -5
[ 2525.425645] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2c3c0 len 32 error 5
[ 2525.425647] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffa80 len 32 error 5
[ 2525.425656] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2525.425711] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2525.592468] XFS (dm-0): Unmounting Filesystem
[ 2525.609374] XFS (dm-0): Mounting V5 Filesystem
[ 2525.681666] XFS (dm-0): Starting recovery (logdev: internal)
[ 2525.694463] XFS (dm-0): Ending recovery (logdev: internal)
[ 2525.696357] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2526.704897] XFS (dm-0): log I/O error -5
[ 2526.704923] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2526.705029] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2526.948328] XFS (dm-0): Unmounting Filesystem
[ 2527.040926] XFS (dm-0): Mounting V5 Filesystem
[ 2527.216170] XFS (dm-0): Starting recovery (logdev: internal)
[ 2527.749123] XFS (dm-0): Ending recovery (logdev: internal)
[ 2527.750067] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2527.755047] XFS (dm-0): log I/O error -5
[ 2527.755045] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2527.755061] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
[ 2527.755080] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2527.755161] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2527.923532] XFS (dm-0): Unmounting Filesystem
[ 2527.940178] XFS (dm-0): Mounting V5 Filesystem
[ 2527.999573] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.014009] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.016171] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2528.021163] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15c0 len 32 error 5
[ 2528.021161] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x13ffe80 len 32 error 5
[ 2528.021247] XFS (dm-0): log I/O error -5
[ 2528.021260] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2528.021319] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2528.261691] XFS (dm-0): Unmounting Filesystem
[ 2528.278383] XFS (dm-0): Mounting V5 Filesystem
[ 2528.337759] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.352023] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.354722] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2528.359910] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2528.359909] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffaa0 len 32 error 5
[ 2528.360016] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2528.360045] XFS (dm-0): log I/O error -5
[ 2528.360016] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2528.360059] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2528.360128] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2528.603702] XFS (dm-0): Unmounting Filesystem
[ 2528.620367] XFS (dm-0): Mounting V5 Filesystem
[ 2528.629620] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.634048] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.635707] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2530.643300] buffer_io_error: 86 callbacks suppressed
[ 2530.643303] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2530.643319] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2530.643323] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2530.643326] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2530.643330] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2530.643333] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2530.643336] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2530.643339] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2530.643342] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2530.643345] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2530.644362] iomap_finish_ioend: 221 callbacks suppressed
[ 2530.644364] dm-0: writeback error on inode 34090876, offset 3014656, sector 22112200
[ 2530.644538] dm-0: writeback error on inode 34427948, offset 786432, sector 22282544
[ 2530.644760] dm-0: writeback error on inode 178971, offset 131072, sector 1225112
[ 2530.645682] dm-0: writeback error on inode 178976, offset 327680, sector 857912
[ 2530.645690] dm-0: writeback error on inode 101264072, offset 884736, sector 63870264
[ 2530.645694] dm-0: writeback error on inode 178976, offset 331776, sector 1089144
[ 2530.645699] dm-0: writeback error on inode 178976, offset 389120, sector 1089256
[ 2530.652881] dm-0: writeback error on inode 67151740, offset 7122944, sector 43064768
[ 2530.653420] dm-0: writeback error on inode 385792, offset 5693440, sector 1185032
[ 2530.653602] dm-0: writeback error on inode 34090849, offset 2056192, sector 21840064
[ 2530.653811] XFS (dm-0): log I/O error -5
[ 2530.653819] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2530.653898] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2530.894975] XFS (dm-0): Unmounting Filesystem
[ 2530.953709] XFS (dm-0): Mounting V5 Filesystem
[ 2531.116725] XFS (dm-0): Starting recovery (logdev: internal)
[ 2531.619363] XFS (dm-0): Ending recovery (logdev: internal)
[ 2531.620537] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2533.630182] XFS (dm-0): log I/O error -5
[ 2533.630204] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2533.630296] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2533.875912] XFS (dm-0): Unmounting Filesystem
[ 2533.944095] XFS (dm-0): Mounting V5 Filesystem
[ 2534.097933] XFS (dm-0): Starting recovery (logdev: internal)
[ 2534.545681] XFS (dm-0): Ending recovery (logdev: internal)
[ 2534.546748] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2534.552289] XFS (dm-0): log I/O error -5
[ 2534.552294] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x86540 len 32 error 5
[ 2534.552317] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2534.552363] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2534.552378] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2534.752003] XFS (dm-0): Unmounting Filesystem
[ 2534.768805] XFS (dm-0): Mounting V5 Filesystem
[ 2534.827769] XFS (dm-0): Starting recovery (logdev: internal)
[ 2534.841175] XFS (dm-0): Ending recovery (logdev: internal)
[ 2534.843039] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2536.849519] iomap_finish_ioend: 102 callbacks suppressed
[ 2536.849523] dm-0: writeback error on inode 1413851, offset 89260032, sector 2095432
[ 2536.850440] buffer_io_error: 38 callbacks suppressed
[ 2536.850442] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2536.850453] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2536.850457] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2536.850460] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2536.850463] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2536.850467] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2536.850470] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2536.850473] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2536.850476] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2536.850479] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2536.851985] dm-0: writeback error on inode 1392803, offset 7856128, sector 2156768
[ 2536.851989] dm-0: writeback error on inode 1358735, offset 32309248, sector 1874480
[ 2536.852481] dm-0: writeback error on inode 102083501, offset 5750784, sector 65056496
[ 2536.857616] dm-0: writeback error on inode 1369530, offset 786432, sector 1936752
[ 2536.858012] dm-0: writeback error on inode 1729116, offset 1294336, sector 2157232
[ 2536.858019] dm-0: writeback error on inode 68591248, offset 29708288, sector 44249952
[ 2536.858025] dm-0: writeback error on inode 35290493, offset 67264512, sector 23190648
[ 2536.858030] dm-0: writeback error on inode 35290493, offset 77107200, sector 23302296
[ 2536.858035] dm-0: writeback error on inode 1369530, offset 868352, sector 1937336
[ 2536.858192] XFS (dm-0): log I/O error -5
[ 2536.858202] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2536.858282] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2537.084788] XFS (dm-0): Unmounting Filesystem
[ 2537.125735] XFS (dm-0): Mounting V5 Filesystem
[ 2537.252751] XFS (dm-0): Starting recovery (logdev: internal)
[ 2537.583254] XFS (dm-0): Ending recovery (logdev: internal)
[ 2537.584277] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2538.595093] XFS (dm-0): log I/O error -5
[ 2538.595110] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2538.595196] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2538.864000] XFS (dm-0): Unmounting Filesystem
[ 2538.905962] XFS (dm-0): Mounting V5 Filesystem
[ 2539.096169] XFS (dm-0): Starting recovery (logdev: internal)
[ 2539.633947] XFS (dm-0): Ending recovery (logdev: internal)
[ 2539.634958] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2541.640724] XFS (dm-0): log I/O error -5
[ 2541.640741] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2541.640802] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2541.935012] XFS (dm-0): Unmounting Filesystem
[ 2541.988684] XFS (dm-0): Mounting V5 Filesystem
[ 2542.116431] XFS (dm-0): Starting recovery (logdev: internal)
[ 2542.441152] XFS (dm-0): Ending recovery (logdev: internal)
[ 2542.442150] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2544.448191] iomap_finish_ioend: 46 callbacks suppressed
[ 2544.448194] dm-0: writeback error on inode 36077872, offset 73859072, sector 24110792
[ 2544.449711] buffer_io_error: 38 callbacks suppressed
[ 2544.449712] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2544.449723] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2544.449727] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2544.449728] dm-0: writeback error on inode 2692401, offset 458752, sector 2775128
[ 2544.449730] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2544.449735] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2544.449738] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2544.449742] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2544.449744] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2544.449748] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2544.449751] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2544.458404] dm-0: writeback error on inode 2391873, offset 466944, sector 2391824
[ 2544.458447] dm-0: writeback error on inode 103160128, offset 937984, sector 65927664
[ 2544.458543] dm-0: writeback error on inode 36054700, offset 33423360, sector 24188272
[ 2544.458549] dm-0: writeback error on inode 103092984, offset 28631040, sector 65845792
[ 2544.458553] dm-0: writeback error on inode 2392584, offset 74911744, sector 2755816
[ 2544.458558] dm-0: writeback error on inode 69726030, offset 21921792, sector 44540328
[ 2544.458563] dm-0: writeback error on inode 2669138, offset 1040384, sector 2801096
[ 2544.458566] dm-0: writeback error on inode 2391873, offset 458752, sector 2391960
[ 2544.458851] XFS (dm-0): log I/O error -5
[ 2544.458858] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2544.458917] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2544.683010] XFS (dm-0): Unmounting Filesystem
[ 2544.739531] XFS (dm-0): Mounting V5 Filesystem
[ 2544.914316] XFS (dm-0): Starting recovery (logdev: internal)
[ 2545.426712] XFS (dm-0): Ending recovery (logdev: internal)
[ 2545.427672] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2546.438765] XFS (dm-0): log I/O error -5
[ 2546.438781] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2546.438842] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2546.727004] XFS (dm-0): Unmounting Filesystem
[ 2546.769608] XFS (dm-0): Mounting V5 Filesystem
[ 2546.936362] XFS (dm-0): Starting recovery (logdev: internal)
[ 2547.383261] XFS (dm-0): Ending recovery (logdev: internal)
[ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
[ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
[ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
[ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2547.389120] XFS (dm-0): log I/O error -5
[ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2547.662818] ------------[ cut here ]------------
[ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
[ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
[ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
[ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
[ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
[ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
[ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
[ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
[ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
[ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
[ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
[ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
[ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
[ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
[ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
[ 2547.662927] Call Trace:
[ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
[ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
[ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
[ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
[ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
[ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
[ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
[ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
[ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
[ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
[ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
[ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
[ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
[ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
[ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
[ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
[ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
[ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
[ 2547.663010] IRQMASK: 0
[ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
[ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
[ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
[ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
[ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
[ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
[ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
[ 2547.663047] --- interrupt: c00
[ 2547.663048] Instruction dump:
[ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
[ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
[ 2547.663062] ---[ end trace 0000000000000000 ]---
[ 2547.663075] XFS (dm-0): Unmounting Filesystem
[ 2547.679882] XFS (dm-0): Mounting V5 Filesystem
[ 2547.733927] XFS (dm-0): Starting recovery (logdev: internal)
[ 2547.745633] XFS (dm-0): Ending recovery (logdev: internal)
[ 2547.748178] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2549.753836] iomap_finish_ioend: 43 callbacks suppressed
[ 2549.753839] dm-0: writeback error on inode 103859326, offset 65249280, sector 66209096
[ 2549.753854] dm-0: writeback error on inode 36109178, offset 6733824, sector 24398488
[ 2549.753859] dm-0: writeback error on inode 70328836, offset 1802240, sector 45162048
[ 2549.753863] dm-0: writeback error on inode 36480701, offset 108093440, sector 24428192
[ 2549.753869] dm-0: writeback error on inode 69618107, offset 22708224, sector 45250888
[ 2549.753873] dm-0: writeback error on inode 36521924, offset 104202240, sector 24543472
[ 2549.753877] dm-0: writeback error on inode 36521924, offset 170885120, sector 24541200
[ 2549.753881] dm-0: writeback error on inode 70357590, offset 3289088, sector 45331584
[ 2549.753885] dm-0: writeback error on inode 103289852, offset 1003520, sector 66297216
[ 2549.754924] XFS (dm-0): log I/O error -5
[ 2549.754950] XFS (dm-0): log I/O error -5
[ 2549.754940] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2549.755035] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2549.756230] buffer_io_error: 38 callbacks suppressed
[ 2549.756233] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2549.756242] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2549.756246] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2549.756249] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2549.756252] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2549.756255] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2549.756259] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2549.756262] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2549.756265] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2549.756268] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2550.021468] XFS (dm-0): Unmounting Filesystem
[ 2550.080673] XFS (dm-0): Mounting V5 Filesystem
[ 2550.270646] XFS (dm-0): Starting recovery (logdev: internal)
[ 2550.760973] XFS (dm-0): Ending recovery (logdev: internal)
[ 2550.762099] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2551.767871] dm-0: writeback error on inode 70406127, offset 124911616, sector 45548040
[ 2551.767885] XFS (dm-0): log I/O error -5
[ 2551.768068] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2551.768140] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2552.001984] XFS (dm-0): Unmounting Filesystem
[ 2552.033560] XFS (dm-0): Mounting V5 Filesystem
[ 2552.202493] XFS (dm-0): Starting recovery (logdev: internal)
[ 2552.629573] XFS (dm-0): Ending recovery (logdev: internal)
[ 2552.630476] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2554.636725] XFS (dm-0): log I/O error -5
[ 2554.636740] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2554.636803] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2554.906255] XFS (dm-0): Unmounting Filesystem
[ 2554.964125] XFS (dm-0): Mounting V5 Filesystem
[ 2555.118190] XFS (dm-0): Starting recovery (logdev: internal)
[ 2555.480176] XFS (dm-0): Ending recovery (logdev: internal)
[ 2555.481202] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2556.487342] XFS (dm-0): log I/O error -5
[ 2556.487360] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2556.487423] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2556.487427] XFS (dm-0): log I/O error -5
[ 2556.487429] XFS (dm-0): log I/O error -5
[ 2556.487431] XFS (dm-0): log I/O error -5
[ 2556.489554] buffer_io_error: 38 callbacks suppressed
[ 2556.489556] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2556.489565] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2556.489569] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2556.489572] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2556.489575] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2556.489579] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2556.489582] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2556.489585] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2556.489588] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2556.489591] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2556.722895] XFS (dm-0): Unmounting Filesystem
[ 2556.781571] XFS (dm-0): Mounting V5 Filesystem
[ 2556.962599] XFS (dm-0): Starting recovery (logdev: internal)
[ 2557.386041] XFS (dm-0): Ending recovery (logdev: internal)
[ 2557.387016] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2558.400557] iomap_finish_ioend: 41 callbacks suppressed
[ 2558.400560] dm-0: writeback error on inode 104621403, offset 393216, sector 66985016
[ 2558.400788] dm-0: writeback error on inode 2973309, offset 3555328, sector 4043448
[ 2558.400797] dm-0: writeback error on inode 3532245, offset 27906048, sector 4008328
[ 2558.400802] dm-0: writeback error on inode 104656344, offset 114626560, sector 66996824
[ 2558.400808] dm-0: writeback error on inode 104621424, offset 77824, sector 66916624
[ 2558.400872] XFS (dm-0): log I/O error -5
[ 2558.400880] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2558.400967] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2558.625070] XFS (dm-0): Unmounting Filesystem
[ 2558.672009] XFS (dm-0): Mounting V5 Filesystem
[ 2558.811196] XFS (dm-0): Starting recovery (logdev: internal)
[ 2559.196329] XFS (dm-0): Ending recovery (logdev: internal)
[ 2559.197243] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2559.202340] XFS (dm-0): log I/O error -5
[ 2559.202345] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3f19480 len 32 error 5
[ 2559.202361] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2559.202420] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2559.432336] XFS (dm-0): Unmounting Filesystem
[ 2559.449047] XFS (dm-0): Mounting V5 Filesystem
[ 2559.490582] XFS (dm-0): Starting recovery (logdev: internal)
[ 2559.499664] XFS (dm-0): Ending recovery (logdev: internal)
[ 2559.501334] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2561.508579] buffer_io_error: 38 callbacks suppressed
[ 2561.508582] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2561.508599] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2561.508603] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2561.508606] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2561.508609] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2561.508613] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2561.508615] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2561.508619] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2561.508622] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2561.508625] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2561.512702] dm-0: writeback error on inode 104703262, offset 147324928, sector 67200392
[ 2561.523022] dm-0: writeback error on inode 37886195, offset 131072, sector 25502904
[ 2561.550302] XFS (dm-0): log I/O error -5
[ 2561.550316] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2561.550376] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2561.550380] XFS (dm-0): log I/O error -5
[ 2561.550382] XFS (dm-0): log I/O error -5
[ 2561.550384] XFS (dm-0): log I/O error -5
[ 2561.789009] XFS (dm-0): Unmounting Filesystem
[ 2561.873689] XFS (dm-0): Mounting V5 Filesystem
[ 2562.009743] XFS (dm-0): Starting recovery (logdev: internal)
[ 2562.358919] XFS (dm-0): Ending recovery (logdev: internal)
[ 2562.359856] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2564.370069] dm-0: writeback error on inode 102464293, offset 2129920, sector 67036960
[ 2564.370076] dm-0: writeback error on inode 37987634, offset 62914560, sector 25414248
[ 2564.370102] dm-0: writeback error on inode 37987634, offset 128565248, sector 25743352
[ 2564.370113] dm-0: writeback error on inode 4143566, offset 1355776, sector 4470760
[ 2564.370119] dm-0: writeback error on inode 37386218, offset 1511424, sector 25184776
[ 2564.370125] dm-0: writeback error on inode 71426166, offset 44986368, sector 46349904
[ 2564.370130] dm-0: writeback error on inode 4233901, offset 22544384, sector 4480616
[ 2564.370135] dm-0: writeback error on inode 37935212, offset 44507136, sector 25651960
[ 2564.370141] dm-0: writeback error on inode 105027811, offset 581632, sector 67474168
[ 2564.370147] dm-0: writeback error on inode 37886177, offset 34197504, sector 25582624
[ 2564.370451] XFS (dm-0): log I/O error -5
[ 2564.370460] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2564.370521] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2564.603752] XFS (dm-0): Unmounting Filesystem
[ 2564.668356] XFS (dm-0): Mounting V5 Filesystem
[ 2564.815810] XFS (dm-0): Starting recovery (logdev: internal)
[ 2565.223316] XFS (dm-0): Ending recovery (logdev: internal)
[ 2565.224235] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2567.231741] buffer_io_error: 22 callbacks suppressed
[ 2567.231744] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2567.231762] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2567.231765] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2567.231768] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2567.231772] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2567.231775] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2567.231778] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2567.231781] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2567.231785] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2567.231788] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2567.442435] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x1c10 len 8 error 5
[ 2567.446460] XFS (dm-0): log I/O error -5
[ 2567.446469] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2567.446526] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2567.469623] XFS (dm-0): Unmounting Filesystem
[ 2567.469653] XFS (dm-0): Failing async write on buffer block 0x421c00. Retrying async write.
[ 2567.469660] XFS (dm-0): Failing async write on buffer block 0x421bd8. Retrying async write.
[ 2567.469663] XFS (dm-0): Failing async write on buffer block 0x2c47ac8. Retrying async write.
[ 2567.469666] XFS (dm-0): Failing async write on buffer block 0x3db7618. Retrying async write.
[ 2567.469669] XFS (dm-0): Failing async write on buffer block 0x3d39bc0. Retrying async write.
[ 2567.469672] XFS (dm-0): Failing async write on buffer block 0x3d39c00. Retrying async write.
[ 2567.469675] XFS (dm-0): Failing async write on buffer block 0x2c46c70. Retrying async write.
[ 2567.469679] XFS (dm-0): Failing async write on buffer block 0x1724220. Retrying async write.
[ 2567.469683] XFS (dm-0): Failing async write on buffer block 0x3d39c08. Retrying async write.
[ 2567.605306] XFS (dm-0): Mounting V5 Filesystem
[ 2567.778964] XFS (dm-0): Starting recovery (logdev: internal)
[ 2568.250498] XFS (dm-0): Ending recovery (logdev: internal)
[ 2568.251454] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2570.257461] iomap_finish_ioend: 93 callbacks suppressed
[ 2570.257465] dm-0: writeback error on inode 71736329, offset 557056, sector 46797816
[ 2570.277368] dm-0: writeback error on inode 101747424, offset 798720, sector 4883144
[ 2570.277386] dm-0: writeback error on inode 4112654, offset 65536, sector 4765656
[ 2570.277392] dm-0: writeback error on inode 71576811, offset 24985600, sector 46689208
[ 2570.277397] dm-0: writeback error on inode 71596503, offset 56389632, sector 46663520
[ 2570.277403] dm-0: writeback error on inode 71596485, offset 70684672, sector 46728408
[ 2570.277408] dm-0: writeback error on inode 105195353, offset 671744, sector 67776440
[ 2570.277414] dm-0: writeback error on inode 71399352, offset 10092544, sector 46274472
[ 2570.277419] dm-0: writeback error on inode 71399352, offset 34680832, sector 46800296
[ 2570.277424] dm-0: writeback error on inode 4500080, offset 18259968, sector 4845360
[ 2570.277599] XFS (dm-0): log I/O error -5
[ 2570.277608] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2570.277680] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2570.509378] XFS (dm-0): Unmounting Filesystem
[ 2570.565428] XFS (dm-0): Mounting V5 Filesystem
[ 2570.757446] XFS (dm-0): Starting recovery (logdev: internal)
[ 2571.291433] XFS (dm-0): Ending recovery (logdev: internal)
[ 2571.292377] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2572.300294] buffer_io_error: 22 callbacks suppressed
[ 2572.300297] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2572.300313] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2572.300317] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2572.300320] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2572.300323] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2572.300326] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2572.300329] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2572.300333] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2572.300336] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2572.300339] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2572.301248] XFS (dm-0): log I/O error -5
[ 2572.301257] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2572.301329] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2572.524483] XFS (dm-0): Unmounting Filesystem
[ 2572.561468] XFS (dm-0): Mounting V5 Filesystem
[ 2572.702822] XFS (dm-0): Starting recovery (logdev: internal)
[ 2573.072094] XFS (dm-0): Ending recovery (logdev: internal)
[ 2573.073038] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2574.091592] XFS (dm-0): log I/O error -5
[ 2574.091610] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2574.091672] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2574.332995] XFS (dm-0): Unmounting Filesystem
[ 2574.368980] XFS (dm-0): Mounting V5 Filesystem
[ 2574.511448] XFS (dm-0): Starting recovery (logdev: internal)
[ 2574.879902] XFS (dm-0): Ending recovery (logdev: internal)
[ 2574.880819] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2576.886825] iomap_finish_ioend: 23 callbacks suppressed
[ 2576.886828] dm-0: writeback error on inode 5086277, offset 23314432, sector 5450480
[ 2576.886844] dm-0: writeback error on inode 71905642, offset 63766528, sector 47007200
[ 2576.886849] dm-0: writeback error on inode 38936684, offset 585728, sector 26394984
[ 2576.886853] dm-0: writeback error on inode 71399332, offset 18481152, sector 47024616
[ 2576.886858] dm-0: writeback error on inode 105806417, offset 839680, sector 68139328
[ 2576.886863] dm-0: writeback error on inode 72124624, offset 13430784, sector 47057936
[ 2576.886867] dm-0: writeback error on inode 72081362, offset 110592, sector 46913208
[ 2576.886872] dm-0: writeback error on inode 3290291, offset 20537344, sector 5361800
[ 2576.886876] dm-0: writeback error on inode 5270796, offset 569344, sector 5267992
[ 2576.886880] dm-0: writeback error on inode 38888467, offset 1638400, sector 26450008
[ 2576.886995] XFS (dm-0): log I/O error -5
[ 2576.887009] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2576.887116] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2577.128229] XFS (dm-0): Unmounting Filesystem
[ 2577.165577] XFS (dm-0): Mounting V5 Filesystem
[ 2577.305996] XFS (dm-0): Starting recovery (logdev: internal)
[ 2577.711552] XFS (dm-0): Ending recovery (logdev: internal)
[ 2577.712499] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2577.717746] XFS (dm-0): log I/O error -5
[ 2577.717743] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
[ 2577.717749] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
[ 2577.717755] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1c10 len 8 error 5
[ 2577.717797] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2577.717899] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2577.720017] buffer_io_error: 38 callbacks suppressed
[ 2577.720019] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2577.720031] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2577.720035] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2577.720038] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2577.720042] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2577.720045] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2577.720048] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2577.720051] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2577.720054] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2577.720057] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2577.963247] XFS (dm-0): Unmounting Filesystem
[ 2577.980236] XFS (dm-0): Mounting V5 Filesystem
[ 2578.029315] XFS (dm-0): Starting recovery (logdev: internal)
[ 2578.040983] XFS (dm-0): Ending recovery (logdev: internal)
[ 2578.042968] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2580.048771] XFS (dm-0): log I/O error -5
[ 2580.048789] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2580.048851] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2580.048855] XFS (dm-0): log I/O error -5
[ 2580.048857] XFS (dm-0): log I/O error -5
[ 2580.048860] XFS (dm-0): log I/O error -5
[ 2580.048862] XFS (dm-0): log I/O error -5
[ 2580.261156] XFS (dm-0): Unmounting Filesystem
[ 2580.317309] XFS (dm-0): Mounting V5 Filesystem
[ 2580.524344] XFS (dm-0): Starting recovery (logdev: internal)
[ 2581.064911] XFS (dm-0): Ending recovery (logdev: internal)
[ 2581.066023] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2582.071952] iomap_finish_ioend: 11 callbacks suppressed
[ 2582.071956] dm-0: writeback error on inode 39381706, offset 417792, sector 26804152
[ 2582.072263] dm-0: writeback error on inode 71802791, offset 50212864, sector 46680008
[ 2582.072837] dm-0: writeback error on inode 104258031, offset 27787264, sector 68423688
[ 2582.072956] dm-0: writeback error on inode 71802753, offset 3035136, sector 47233248
[ 2582.073108] dm-0: writeback error on inode 105687197, offset 262144, sector 67937776
[ 2582.073118] dm-0: writeback error on inode 105687197, offset 19116032, sector 68428184
[ 2582.073126] dm-0: writeback error on inode 5290517, offset 897024, sector 5292776
[ 2582.073134] dm-0: writeback error on inode 105828662, offset 162582528, sector 47243656
[ 2582.073141] dm-0: writeback error on inode 5275003, offset 4661248, sector 5660280
[ 2582.073149] dm-0: writeback error on inode 104258031, offset 54648832, sector 68428352
[ 2582.073301] XFS (dm-0): log I/O error -5
[ 2582.073312] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2582.073422] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2582.329214] XFS (dm-0): Unmounting Filesystem
[ 2582.417520] XFS (dm-0): Mounting V5 Filesystem
[ 2582.553350] XFS (dm-0): Starting recovery (logdev: internal)
[ 2582.842465] XFS (dm-0): Ending recovery (logdev: internal)
[ 2582.843363] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2584.848950] XFS (dm-0): log I/O error -5
[ 2584.848969] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2584.849030] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2584.849033] XFS (dm-0): log I/O error -5
[ 2584.849036] XFS (dm-0): log I/O error -5
[ 2584.849038] XFS (dm-0): log I/O error -5
[ 2584.849040] XFS (dm-0): log I/O error -5
[ 2584.851337] buffer_io_error: 38 callbacks suppressed
[ 2584.851340] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2584.851351] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2584.851355] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2584.851358] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2584.851362] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2584.851365] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2584.851368] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2584.851371] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2584.851374] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2584.851377] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2585.125926] XFS (dm-0): Unmounting Filesystem
[ 2585.165412] XFS (dm-0): Mounting V5 Filesystem
[ 2585.354102] XFS (dm-0): Starting recovery (logdev: internal)
[ 2585.860897] XFS (dm-0): Ending recovery (logdev: internal)
[ 2585.861828] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2586.870178] XFS (dm-0): log I/O error -5
[ 2586.870200] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2586.870286] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2587.095054] XFS (dm-0): Unmounting Filesystem
[ 2587.134976] XFS (dm-0): Mounting V5 Filesystem
[ 2587.275993] XFS (dm-0): Starting recovery (logdev: internal)
[ 2587.537406] XFS (dm-0): Ending recovery (logdev: internal)
[ 2587.538293] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2587.543301] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2587.543302] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2587.543303] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
[ 2587.543299] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x40f09c0 len 32 error 5
[ 2587.543358] XFS (dm-0): page discard on page 000000008449fa28, inode 0x64c7ea4, pos 589824.
[ 2587.543371] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2587.543429] XFS (dm-0): log I/O error -5
[ 2587.543450] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2587.543514] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2587.744847] XFS (dm-0): Unmounting Filesystem
[ 2587.814615] XFS (dm-0): Mounting V5 Filesystem
[ 2587.844735] XFS (dm-0): Starting recovery (logdev: internal)
[ 2587.853639] XFS (dm-0): Ending recovery (logdev: internal)
[ 2587.855380] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2588.861351] iomap_finish_ioend: 17 callbacks suppressed
[ 2588.861355] dm-0: writeback error on inode 5669593, offset 1048576, sector 27217304
[ 2588.861747] dm-0: writeback error on inode 72173797, offset 655360, sector 47498000
[ 2588.863767] dm-0: writeback error on inode 5686085, offset 81920, sector 5856320
[ 2588.864118] XFS (dm-0): log I/O error -5
[ 2588.864129] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2588.864192] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2589.087112] XFS (dm-0): Unmounting Filesystem
[ 2589.126196] XFS (dm-0): Mounting V5 Filesystem
[ 2589.256674] XFS (dm-0): Starting recovery (logdev: internal)
[ 2589.542959] XFS (dm-0): Ending recovery (logdev: internal)
[ 2589.544017] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2590.551430] buffer_io_error: 54 callbacks suppressed
[ 2590.551434] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2590.551450] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2590.551454] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2590.551457] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2590.551461] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2590.551464] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2590.551467] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2590.551470] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2590.551473] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2590.551476] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2590.776047] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2590.777568] dm-0: writeback error on inode 39082395, offset 2023424, sector 26498480
[ 2590.777572] dm-0: writeback error on inode 4054818, offset 58073088, sector 5992088
[ 2590.777578] dm-0: writeback error on inode 4054818, offset 173793280, sector 5998104
[ 2590.777583] dm-0: writeback error on inode 4054818, offset 178622464, sector 5999312
[ 2590.777587] dm-0: writeback error on inode 39629723, offset 909312, sector 27047544
[ 2590.777592] dm-0: writeback error on inode 5686139, offset 26587136, sector 6010928
[ 2590.777596] dm-0: writeback error on inode 72492761, offset 217088, sector 47563032
[ 2590.777692] XFS (dm-0): log I/O error -5
[ 2590.777699] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2590.777757] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2590.803098] XFS (dm-0): Unmounting Filesystem
[ 2590.803127] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2590.803136] XFS (dm-0): Failing async write on buffer block 0x171cf30. Retrying async write.
[ 2590.803140] XFS (dm-0): Failing async write on buffer block 0x171d098. Retrying async write.
[ 2590.803143] XFS (dm-0): Failing async write on buffer block 0x13ffe28. Retrying async write.
[ 2590.803147] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
[ 2590.803151] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
[ 2590.803155] XFS (dm-0): Failing async write on buffer block 0x18. Retrying async write.
[ 2590.803158] XFS (dm-0): Failing async write on buffer block 0x28. Retrying async write.
[ 2590.803161] XFS (dm-0): Failing async write on buffer block 0x20. Retrying async write.
[ 2590.898854] XFS (dm-0): Mounting V5 Filesystem
[ 2591.015030] XFS (dm-0): Starting recovery (logdev: internal)
[ 2591.269532] XFS (dm-0): Ending recovery (logdev: internal)
[ 2591.270553] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2592.278892] XFS (dm-0): log I/O error -5
[ 2592.278909] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2592.279024] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2592.534075] XFS (dm-0): Unmounting Filesystem
[ 2592.573250] XFS (dm-0): Mounting V5 Filesystem
[ 2592.700931] XFS (dm-0): Starting recovery (logdev: internal)
[ 2593.029802] XFS (dm-0): Ending recovery (logdev: internal)
[ 2593.030847] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2594.233003] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2594.233514] iomap_finish_ioend: 21 callbacks suppressed
[ 2594.233516] dm-0: writeback error on inode 38125892, offset 651264, sector 27377000
[ 2594.233526] dm-0: writeback error on inode 36841118, offset 83578880, sector 27377256
[ 2594.233533] dm-0: writeback error on inode 3267698, offset 7778304, sector 6173120
[ 2594.233540] dm-0: writeback error on inode 38125940, offset 524288, sector 27335560
[ 2594.233546] dm-0: writeback error on inode 38125940, offset 1290240, sector 27335264
[ 2594.233553] dm-0: writeback error on inode 3267673, offset 74072064, sector 6173256
[ 2594.233677] XFS (dm-0): log I/O error -5
[ 2594.233688] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2594.233789] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2594.250213] XFS (dm-0): Failing async write on buffer block 0x1869620. Retrying async write.
[ 2594.250220] XFS (dm-0): Failing async write on buffer block 0x1869600. Retrying async write.
[ 2594.259160] XFS (dm-0): Unmounting Filesystem
[ 2594.259192] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2594.259197] XFS (dm-0): Failing async write on buffer block 0x40a8590. Retrying async write.
[ 2594.259201] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
[ 2594.259205] XFS (dm-0): Failing async write on buffer block 0x19cb4a8. Retrying async write.
[ 2594.259212] XFS (dm-0): Failing async write on buffer block 0x598438. Retrying async write.
[ 2594.259215] XFS (dm-0): Failing async write on buffer block 0x2d560a0. Retrying async write.
[ 2594.259221] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
[ 2594.358902] XFS (dm-0): Mounting V5 Filesystem
[ 2594.463445] XFS (dm-0): Starting recovery (logdev: internal)
[ 2594.706900] XFS (dm-0): Ending recovery (logdev: internal)
[ 2594.707815] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2596.713662] dm-0: writeback error on inode 6257128, offset 716800, sector 6320832
[ 2596.713668] dm-0: writeback error on inode 3509159, offset 10272768, sector 3534256
[ 2596.713682] dm-0: writeback error on inode 105936492, offset 1105920, sector 68385744
[ 2596.713687] dm-0: writeback error on inode 72828680, offset 1441792, sector 47710184
[ 2596.713805] XFS (dm-0): log I/O error -5
[ 2596.713816] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2596.713888] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2596.716177] buffer_io_error: 38 callbacks suppressed
[ 2596.716179] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2596.716188] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2596.716191] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2596.716195] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2596.716198] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2596.716201] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2596.716204] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2596.716207] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2596.716210] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2596.716213] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2596.944807] XFS (dm-0): Unmounting Filesystem
[ 2596.987515] XFS (dm-0): Mounting V5 Filesystem
[ 2597.146921] XFS (dm-0): Starting recovery (logdev: internal)
[ 2597.506121] XFS (dm-0): Ending recovery (logdev: internal)
[ 2597.507036] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2598.694283] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
[ 2598.695795] XFS (dm-0): log I/O error -5
[ 2598.695803] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2598.695860] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2598.695864] XFS (dm-0): log I/O error -5
[ 2598.695866] XFS (dm-0): log I/O error -5
[ 2598.695868] XFS (dm-0): log I/O error -5
[ 2598.695870] XFS (dm-0): log I/O error -5
[ 2598.721263] XFS (dm-0): Unmounting Filesystem
[ 2598.721294] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2598.721301] XFS (dm-0): Failing async write on buffer block 0x14082c0. Retrying async write.
[ 2598.721305] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
[ 2598.721308] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
[ 2598.721311] XFS (dm-0): Failing async write on buffer block 0x3c032d0. Retrying async write.
[ 2598.721314] XFS (dm-0): Failing async write on buffer block 0x1869810. Retrying async write.
[ 2598.721317] XFS (dm-0): Failing async write on buffer block 0x3c03808. Retrying async write.
[ 2598.721320] XFS (dm-0): Failing async write on buffer block 0x4092918. Retrying async write.
[ 2598.721324] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
[ 2598.819903] XFS (dm-0): Mounting V5 Filesystem
[ 2598.926017] XFS (dm-0): Starting recovery (logdev: internal)
[ 2599.185692] XFS (dm-0): Ending recovery (logdev: internal)
[ 2599.186561] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2599.232676] XFS (dm-0): log I/O error -5
[ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x358b80 len 32 error 5
[ 2599.232693] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2599.232755] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2599.463847] XFS (dm-0): Unmounting Filesystem
[ 2599.480839] XFS (dm-0): Mounting V5 Filesystem
[ 2599.512736] XFS (dm-0): Starting recovery (logdev: internal)
[ 2599.520518] XFS (dm-0): Ending recovery (logdev: internal)
[ 2599.522021] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2601.544049] iomap_finish_ioend: 13 callbacks suppressed
[ 2601.544052] dm-0: writeback error on inode 6344500, offset 77398016, sector 6522176
[ 2601.693208] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
[ 2601.695457] dm-0: writeback error on inode 73002350, offset 69632, sector 47941664
[ 2601.695637] dm-0: writeback error on inode 73002350, offset 180224, sector 47943056
[ 2601.695645] dm-0: writeback error on inode 6211174, offset 1847296, sector 6452984
[ 2601.695649] dm-0: writeback error on inode 72952535, offset 57147392, sector 47919256
[ 2601.695653] dm-0: writeback error on inode 72967749, offset 46862336, sector 47943344
[ 2601.695657] dm-0: writeback error on inode 106150432, offset 524288, sector 68803664
[ 2601.695661] dm-0: writeback error on inode 106150432, offset 17850368, sector 69367752
[ 2601.695665] dm-0: writeback error on inode 106502774, offset 84537344, sector 69368088
[ 2601.695670] dm-0: writeback error on inode 6447171, offset 106463232, sector 6626344
[ 2601.695792] XFS (dm-0): log I/O error -5
[ 2601.695799] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2601.695857] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2601.725357] XFS (dm-0): Unmounting Filesystem
[ 2601.725391] XFS (dm-0): Failing async write on buffer block 0x2d96240. Retrying async write.
[ 2601.725396] XFS (dm-0): Failing async write on buffer block 0x3fca268. Retrying async write.
[ 2601.725401] XFS (dm-0): Failing async write on buffer block 0x41f6820. Retrying async write.
[ 2601.725404] XFS (dm-0): Failing async write on buffer block 0x41f6800. Retrying async write.
[ 2601.725407] XFS (dm-0): Failing async write on buffer block 0x623aa0. Retrying async write.
[ 2601.725410] XFS (dm-0): Failing async write on buffer block 0x623a80. Retrying async write.
[ 2601.725413] XFS (dm-0): Failing async write on buffer block 0x280a380. Retrying async write.
[ 2601.725416] XFS (dm-0): Failing async write on buffer block 0x2d96280. Retrying async write.
[ 2601.725419] XFS (dm-0): Failing async write on buffer block 0x196c938. Retrying async write.
[ 2601.880399] XFS (dm-0): Mounting V5 Filesystem
[ 2602.048576] XFS (dm-0): Starting recovery (logdev: internal)
[ 2602.486670] XFS (dm-0): Ending recovery (logdev: internal)
[ 2602.487581] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2603.494852] buffer_io_error: 54 callbacks suppressed
[ 2603.494855] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2603.494873] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2603.494877] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2603.494880] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2603.494883] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2603.494887] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2603.494889] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2603.494893] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2603.494896] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2603.494899] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2603.721652] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2603.723184] XFS (dm-0): log I/O error -5
[ 2603.723194] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2603.723276] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2603.750782] XFS (dm-0): Unmounting Filesystem
[ 2603.750814] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2603.750820] XFS (dm-0): Failing async write on buffer block 0x1920760. Retrying async write.
[ 2603.750823] XFS (dm-0): Failing async write on buffer block 0x3c02ea0. Retrying async write.
[ 2603.750827] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
[ 2603.750840] XFS (dm-0): Failing async write on buffer block 0x41bb2a0. Retrying async write.
[ 2603.750844] XFS (dm-0): Failing async write on buffer block 0x14155a0. Retrying async write.
[ 2603.750847] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
[ 2603.750850] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
[ 2603.750859] XFS (dm-0): Failing async write on buffer block 0x42058e0. Retrying async write.
[ 2603.911355] XFS (dm-0): Mounting V5 Filesystem
[ 2604.039283] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.316945] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.317844] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323032] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323042] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323088] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323104] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323138] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323169] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323188] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323247] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.324979] XFS (dm-0): log I/O error -5
[ 2604.324999] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2604.325082] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2604.554164] XFS (dm-0): Unmounting Filesystem
[ 2604.571381] XFS (dm-0): Mounting V5 Filesystem
[ 2604.613536] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.620050] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.621596] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.627298] XFS (dm-0): log I/O error -5
[ 2604.627307] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x1869600 len 32 error 5
[ 2604.627334] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2604.627438] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2604.813933] XFS (dm-0): Unmounting Filesystem
[ 2604.830589] XFS (dm-0): Mounting V5 Filesystem
[ 2604.875920] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.884974] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.886426] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.932799] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x174bee0 len 32 error 5
[ 2604.932802] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
[ 2604.932803] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2604.932900] XFS (dm-0): log I/O error -5
[ 2604.932915] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2604.932974] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2605.164824] XFS (dm-0): Unmounting Filesystem
[ 2605.181637] XFS (dm-0): Mounting V5 Filesystem
[ 2605.225897] XFS (dm-0): Starting recovery (logdev: internal)
[ 2605.232410] XFS (dm-0): Ending recovery (logdev: internal)
[ 2605.234452] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2607.239943] iomap_finish_ioend: 40 callbacks suppressed
[ 2607.239946] dm-0: writeback error on inode 40220526, offset 31604736, sector 27637824
[ 2607.239960] dm-0: writeback error on inode 40021920, offset 29814784, sector 27875352
[ 2607.239965] dm-0: writeback error on inode 104667118, offset 105447424, sector 69504936
[ 2607.239969] dm-0: writeback error on inode 104667118, offset 171417600, sector 69614576
[ 2607.239974] dm-0: writeback error on inode 6344484, offset 25358336, sector 6857952
[ 2607.240036] XFS (dm-0): log I/O error -5
[ 2607.240049] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2607.240137] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2607.515399] XFS (dm-0): Unmounting Filesystem
[ 2607.560484] XFS (dm-0): Mounting V5 Filesystem
[ 2607.709693] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.076848] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.077772] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
[ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2608.082758] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082810] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082862] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082882] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2608.082962] XFS (dm-0): log I/O error -5
[ 2608.082977] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2608.083049] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2608.083059] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2608.283930] XFS (dm-0): Unmounting Filesystem
[ 2608.301209] XFS (dm-0): Mounting V5 Filesystem
[ 2608.359920] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.370818] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.372848] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2608.377740] XFS (dm-0): log I/O error -5
[ 2608.377743] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404288 len 8 error 5
[ 2608.377745] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3fc5f60 len 32 error 5
[ 2608.377761] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2608.377823] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2608.603843] XFS (dm-0): Unmounting Filesystem
[ 2608.620761] XFS (dm-0): Mounting V5 Filesystem
[ 2608.682744] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.695059] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.696923] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2609.704216] buffer_io_error: 102 callbacks suppressed
[ 2609.704219] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2609.704235] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2609.704239] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2609.704242] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2609.704246] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2609.704249] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2609.704252] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2609.704255] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2609.704258] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2609.704261] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2609.704971] dm-0: writeback error on inode 73088154, offset 87687168, sector 48133584
[ 2609.868626] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x48 len 8 error 5
[ 2609.869085] dm-0: writeback error on inode 40295105, offset 1048576, sector 27860880
[ 2609.869194] dm-0: writeback error on inode 106528698, offset 983040, sector 68387864
[ 2609.869443] dm-0: writeback error on inode 40295105, offset 479232, sector 27704168
[ 2609.869452] dm-0: writeback error on inode 39669888, offset 46268416, sector 28022120
[ 2609.869639] XFS (dm-0): log I/O error -5
[ 2609.869650] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2609.869750] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2609.897807] XFS (dm-0): Unmounting Filesystem
[ 2609.897839] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2609.897851] XFS (dm-0): Failing async write on buffer block 0x1404c08. Retrying async write.
[ 2609.897855] XFS (dm-0): Failing async write on buffer block 0x17ced58. Retrying async write.
[ 2609.897860] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
[ 2609.897863] XFS (dm-0): Failing async write on buffer block 0x5937e8. Retrying async write.
[ 2609.897866] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
[ 2609.897882] XFS (dm-0): Failing async write on buffer block 0x657400. Retrying async write.
[ 2609.897886] XFS (dm-0): Failing async write on buffer block 0x689620. Retrying async write.
[ 2609.897888] XFS (dm-0): Failing async write on buffer block 0x689600. Retrying async write.
[ 2610.051099] XFS (dm-0): Mounting V5 Filesystem
[ 2610.163869] XFS (dm-0): Starting recovery (logdev: internal)
[ 2610.426613] XFS (dm-0): Ending recovery (logdev: internal)
[ 2610.427541] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2610.435505] XFS (dm-0): Unmounting Filesystem
[ 2610.546863] XFS (sdb1): Unmounting Filesystem
[ 2611.152679] run fstests generic/476 at 2022-02-15 15:43:38
^ permalink raw reply
* Re: [PATCH v2 1/2] crypto: vmx: Turn CRYPTO_DEV_VMX_ENCRYPT into tristate
From: Petr Vorel @ 2022-02-16 7:24 UTC (permalink / raw)
To: linux-crypto
Cc: Herbert Xu, Nicolai Stange, linux-kbuild, Nayna Jain,
Paulo Flabiano Smorigo, leitao, linuxppc-dev
In-Reply-To: <20220215185936.15576-2-pvorel@suse.cz>
Hi,
I kept CRYPTO_DEV_VMX_ENCRYPT in drivers/crypto/vmx/Kconfig,
but maybe I should have moved it into drivers/crypto/Kconfig.
It's not clear what is preferred.
Kind regards,
Petr
^ permalink raw reply
* [PATCH v4 0/6] Improve KVM's interaction with CPU hotplug
From: Chao Gao @ 2022-02-16 3:15 UTC (permalink / raw)
To: seanjc, maz, kvm, pbonzini, kevin.tian, tglx
Cc: Thomas Richter, Catalin Marinas, David Hildenbrand,
Maciej S. Szmigiero, Dave Hansen, x86, linux-mips, Paul Mackerras,
H. Peter Anvin, Wanpeng Li, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, kvmarm, linux-s390, Jia He, Janosch Frank,
Anup Patel, Joerg Roedel, Huacai Chen, linux-riscv,
Aleksandar Markovic, Ingo Molnar, Darrick J. Wong, Palmer Dabbelt,
Christian Borntraeger, Qi Liu, Chao Gao, Andrew Jones, Albert Ou,
Vasily Gorbik, Suzuki K Poulose, Heiko Carstens, John Garry,
Nicholas Piggin, Nathan Chancellor, Eric Auger, Tom Zanussi,
Borislav Petkov, Paul Walmsley, Atish Patra, Sumanth Korikkar,
Alexandru Elisei, linux-arm-kernel, Jim Mattson, Juergen Gross,
Shaokun Zhang, Thomas Bogendoerfer, Hector Martin,
Nick Desaulniers, linux-kernel, Bharata B Rao, James Morse,
Sven Schnelle, kvm-riscv, Vitaly Kuznetsov, linuxppc-dev
Changes from v3->v4:
- rebased to the lastest kvm/next branch.
- add Sean's reviewed-by tags
- add Marc's patch to simplify ARM's cpu hotplug logic in KVM
Changes from v2->v3:
- rebased to the latest kvm/next branch.
- patch 1: rename {svm,vmx}_check_processor_compat to follow the name
convention
- patch 3: newly added to provide more information when hardware enabling
fails
- patch 4: reset hardware_enable_failed if hardware enabling fails. And
remove redundent kernel log.
- patch 5: add a pr_err() for setup_vmcs_config() path.
Changes from v1->v2: (all comments/suggestions on v1 are from Sean, thanks)
- Merged v1's patch 2 into patch 1, and v1's patch 5 into patch 6.
- Use static_call for check_processor_compatibility().
- Generate patch 2 with "git revert" and do manual changes based on that.
- Loosen the WARN_ON() in kvm_arch_check_processor_compat() instead of
removing it.
- KVM always prevent incompatible CPUs from being brought up regardless of
running VMs.
- Use pr_warn instead of pr_info to emit logs when KVM finds offending
CPUs.
KVM registers its CPU hotplug callback to CPU starting section. And in the
callback, KVM enables hardware virtualization on hotplugged CPUs if any VM
is running on existing CPUs.
There are two problems in the process:
1. KVM doesn't do compatibility checks before enabling hardware
virtualization on hotplugged CPUs. This may cause #GP if VMX isn't
supported or vmentry failure if some in-use VMX features are missing on
hotplugged CPUs. Both break running VMs.
2. Callbacks in CPU STARTING section cannot fail. So, even if KVM finds
some incompatible CPUs, its callback cannot block CPU hotplug.
This series improves KVM's interaction with CPU hotplug to avoid
incompatible CPUs breaking running VMs. Following changes are made:
1. move KVM's CPU hotplug callback to ONLINE section (suggested by Thomas)
2. do compatibility checks on hotplugged CPUs.
3. abort onlining incompatible CPUs
This series is a follow-up to the discussion about KVM and CPU hotplug
https://lore.kernel.org/lkml/3d3296f0-9245-40f9-1b5a-efffdb082de9@redhat.com/T/
Note: this series is tested only on Intel systems.
Chao Gao (4):
KVM: x86: Move check_processor_compatibility from init ops to runtime
ops
Partially revert "KVM: Pass kvm_init()'s opaque param to additional
arch funcs"
KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section
KVM: Do compatibility checks on hotplugged CPUs
Marc Zyngier (1):
KVM: arm64: Simplify the CPUHP logic
Sean Christopherson (1):
KVM: Provide more information in kernel log if hardware enabling fails
arch/arm64/kvm/arch_timer.c | 27 ++++-------
arch/arm64/kvm/arm.c | 6 ++-
arch/arm64/kvm/vgic/vgic-init.c | 19 +-------
arch/mips/kvm/mips.c | 2 +-
arch/powerpc/kvm/powerpc.c | 2 +-
arch/riscv/kvm/main.c | 2 +-
arch/s390/kvm/kvm-s390.c | 2 +-
arch/x86/include/asm/kvm-x86-ops.h | 1 +
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/svm/svm.c | 4 +-
arch/x86/kvm/vmx/evmcs.c | 2 +-
arch/x86/kvm/vmx/evmcs.h | 2 +-
arch/x86/kvm/vmx/vmx.c | 22 +++++----
arch/x86/kvm/x86.c | 16 +++++--
include/kvm/arm_arch_timer.h | 4 ++
include/kvm/arm_vgic.h | 4 ++
include/linux/cpuhotplug.h | 5 +-
include/linux/kvm_host.h | 2 +-
virt/kvm/kvm_main.c | 73 +++++++++++++++++++-----------
19 files changed, 107 insertions(+), 90 deletions(-)
--
2.25.1
^ permalink raw reply
* Re: [PATCH v4 2/6] Partially revert "KVM: Pass kvm_init()'s opaque param to additional arch funcs"
From: Anup Patel @ 2022-02-16 3:25 UTC (permalink / raw)
To: Chao Gao
Cc: x86, Wanpeng Li, KVM General, David Hildenbrand, Catalin Marinas,
Paul Walmsley, linux-mips, Paul Mackerras, H. Peter Anvin,
Alexander Gordeev, Claudio Imbrenda, Will Deacon,
Maciej S. Szmigiero, linux-s390, Janosch Frank, Joerg Roedel,
Huacai Chen, linux-riscv, kvmarm, Dave Hansen,
Aleksandar Markovic, Ingo Molnar, Palmer Dabbelt,
Christian Borntraeger, kevin.tian, Albert Ou, Vasily Gorbik,
Suzuki K Poulose, Heiko Carstens, Nicholas Piggin,
Borislav Petkov, Atish Patra, Thomas Gleixner, Alexandru Elisei,
linux-arm-kernel, Jim Mattson, Thomas Bogendoerfer,
Sean Christopherson, Nick Desaulniers,
linux-kernel@vger.kernel.org List, Bharata B Rao, James Morse,
Sven Schnelle, kvm-riscv, Marc Zyngier, Paolo Bonzini,
Vitaly Kuznetsov, linuxppc-dev
In-Reply-To: <20220216031528.92558-3-chao.gao@intel.com>
On Wed, Feb 16, 2022 at 8:46 AM Chao Gao <chao.gao@intel.com> wrote:
>
> This partially reverts commit b99040853738 ("KVM: Pass kvm_init()'s opaque
> param to additional arch funcs") remove opaque from
> kvm_arch_check_processor_compat because no one uses this opaque now.
> Address conflicts for ARM (due to file movement) and manually handle RISC-V
> which comes after the commit.
>
> And changes about kvm_arch_hardware_setup() in original commit are still
> needed so they are not reverted.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Sean Christopherson <seanjc@google.com>
For KVM RISC-V:
Acked-by: Anup Patel <anup@brainfault.org>
Regards,
Anup
> ---
> arch/arm64/kvm/arm.c | 2 +-
> arch/mips/kvm/mips.c | 2 +-
> arch/powerpc/kvm/powerpc.c | 2 +-
> arch/riscv/kvm/main.c | 2 +-
> arch/s390/kvm/kvm-s390.c | 2 +-
> arch/x86/kvm/x86.c | 2 +-
> include/linux/kvm_host.h | 2 +-
> virt/kvm/kvm_main.c | 16 +++-------------
> 8 files changed, 10 insertions(+), 20 deletions(-)
>
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index ecc5958e27fe..0165cf3aac3a 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -73,7 +73,7 @@ int kvm_arch_hardware_setup(void *opaque)
> return 0;
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> return 0;
> }
> diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c
> index a25e0b73ee70..092d09fb6a7e 100644
> --- a/arch/mips/kvm/mips.c
> +++ b/arch/mips/kvm/mips.c
> @@ -140,7 +140,7 @@ int kvm_arch_hardware_setup(void *opaque)
> return 0;
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> return 0;
> }
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 2ad0ccd202d5..30c817f3fa0c 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -423,7 +423,7 @@ int kvm_arch_hardware_setup(void *opaque)
> return 0;
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> return kvmppc_core_check_processor_compat();
> }
> diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
> index 2e5ca43c8c49..992877e78393 100644
> --- a/arch/riscv/kvm/main.c
> +++ b/arch/riscv/kvm/main.c
> @@ -20,7 +20,7 @@ long kvm_arch_dev_ioctl(struct file *filp,
> return -EINVAL;
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> return 0;
> }
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 577f1ead6a51..0053b81c6b02 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -252,7 +252,7 @@ int kvm_arch_hardware_enable(void)
> return 0;
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> return 0;
> }
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 9b484ed61f37..ffb88f0b7265 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -11509,7 +11509,7 @@ void kvm_arch_hardware_unsetup(void)
> static_call(kvm_x86_hardware_unsetup)();
> }
>
> -int kvm_arch_check_processor_compat(void *opaque)
> +int kvm_arch_check_processor_compat(void)
> {
> struct cpuinfo_x86 *c = &cpu_data(smp_processor_id());
>
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index f11039944c08..2ad78e729bf7 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1413,7 +1413,7 @@ int kvm_arch_hardware_enable(void);
> void kvm_arch_hardware_disable(void);
> int kvm_arch_hardware_setup(void *opaque);
> void kvm_arch_hardware_unsetup(void);
> -int kvm_arch_check_processor_compat(void *opaque);
> +int kvm_arch_check_processor_compat(void);
> int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
> bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
> int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 83c57bcc6eb6..ee47d33d69e1 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -5643,22 +5643,14 @@ void kvm_unregister_perf_callbacks(void)
> }
> #endif
>
> -struct kvm_cpu_compat_check {
> - void *opaque;
> - int *ret;
> -};
> -
> -static void check_processor_compat(void *data)
> +static void check_processor_compat(void *rtn)
> {
> - struct kvm_cpu_compat_check *c = data;
> -
> - *c->ret = kvm_arch_check_processor_compat(c->opaque);
> + *(int *)rtn = kvm_arch_check_processor_compat();
> }
>
> int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
> struct module *module)
> {
> - struct kvm_cpu_compat_check c;
> int r;
> int cpu;
>
> @@ -5686,10 +5678,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
> if (r < 0)
> goto out_free_1;
>
> - c.ret = &r;
> - c.opaque = opaque;
> for_each_online_cpu(cpu) {
> - smp_call_function_single(cpu, check_processor_compat, &c, 1);
> + smp_call_function_single(cpu, check_processor_compat, &r, 1);
> if (r < 0)
> goto out_free_2;
> }
> --
> 2.25.1
>
^ permalink raw reply
* [PATCH v4 2/6] Partially revert "KVM: Pass kvm_init()'s opaque param to additional arch funcs"
From: Chao Gao @ 2022-02-16 3:15 UTC (permalink / raw)
To: seanjc, maz, kvm, pbonzini, kevin.tian, tglx
Cc: x86, Wanpeng Li, David Hildenbrand, Paul Walmsley, linux-mips,
Paul Mackerras, H. Peter Anvin, Alexander Gordeev,
Claudio Imbrenda, Will Deacon, Maciej S. Szmigiero, linux-s390,
Janosch Frank, Anup Patel, Joerg Roedel, Huacai Chen, linux-riscv,
kvmarm, Dave Hansen, Aleksandar Markovic, Ingo Molnar,
Catalin Marinas, Palmer Dabbelt, Christian Borntraeger, Chao Gao,
Albert Ou, Vasily Gorbik, Suzuki K Poulose, Heiko Carstens,
Nicholas Piggin, Borislav Petkov, Atish Patra, Alexandru Elisei,
linux-arm-kernel, Jim Mattson, Thomas Bogendoerfer,
Nick Desaulniers, linux-kernel, Bharata B Rao, James Morse,
Sven Schnelle, kvm-riscv, Vitaly Kuznetsov, linuxppc-dev
In-Reply-To: <20220216031528.92558-1-chao.gao@intel.com>
This partially reverts commit b99040853738 ("KVM: Pass kvm_init()'s opaque
param to additional arch funcs") remove opaque from
kvm_arch_check_processor_compat because no one uses this opaque now.
Address conflicts for ARM (due to file movement) and manually handle RISC-V
which comes after the commit.
And changes about kvm_arch_hardware_setup() in original commit are still
needed so they are not reverted.
Signed-off-by: Chao Gao <chao.gao@intel.com>
Reviewed-by: Sean Christopherson <seanjc@google.com>
---
arch/arm64/kvm/arm.c | 2 +-
arch/mips/kvm/mips.c | 2 +-
arch/powerpc/kvm/powerpc.c | 2 +-
arch/riscv/kvm/main.c | 2 +-
arch/s390/kvm/kvm-s390.c | 2 +-
arch/x86/kvm/x86.c | 2 +-
include/linux/kvm_host.h | 2 +-
virt/kvm/kvm_main.c | 16 +++-------------
8 files changed, 10 insertions(+), 20 deletions(-)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index ecc5958e27fe..0165cf3aac3a 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -73,7 +73,7 @@ int kvm_arch_hardware_setup(void *opaque)
return 0;
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
return 0;
}
diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c
index a25e0b73ee70..092d09fb6a7e 100644
--- a/arch/mips/kvm/mips.c
+++ b/arch/mips/kvm/mips.c
@@ -140,7 +140,7 @@ int kvm_arch_hardware_setup(void *opaque)
return 0;
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 2ad0ccd202d5..30c817f3fa0c 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -423,7 +423,7 @@ int kvm_arch_hardware_setup(void *opaque)
return 0;
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
return kvmppc_core_check_processor_compat();
}
diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
index 2e5ca43c8c49..992877e78393 100644
--- a/arch/riscv/kvm/main.c
+++ b/arch/riscv/kvm/main.c
@@ -20,7 +20,7 @@ long kvm_arch_dev_ioctl(struct file *filp,
return -EINVAL;
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
return 0;
}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 577f1ead6a51..0053b81c6b02 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -252,7 +252,7 @@ int kvm_arch_hardware_enable(void)
return 0;
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
return 0;
}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 9b484ed61f37..ffb88f0b7265 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -11509,7 +11509,7 @@ void kvm_arch_hardware_unsetup(void)
static_call(kvm_x86_hardware_unsetup)();
}
-int kvm_arch_check_processor_compat(void *opaque)
+int kvm_arch_check_processor_compat(void)
{
struct cpuinfo_x86 *c = &cpu_data(smp_processor_id());
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index f11039944c08..2ad78e729bf7 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1413,7 +1413,7 @@ int kvm_arch_hardware_enable(void);
void kvm_arch_hardware_disable(void);
int kvm_arch_hardware_setup(void *opaque);
void kvm_arch_hardware_unsetup(void);
-int kvm_arch_check_processor_compat(void *opaque);
+int kvm_arch_check_processor_compat(void);
int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 83c57bcc6eb6..ee47d33d69e1 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -5643,22 +5643,14 @@ void kvm_unregister_perf_callbacks(void)
}
#endif
-struct kvm_cpu_compat_check {
- void *opaque;
- int *ret;
-};
-
-static void check_processor_compat(void *data)
+static void check_processor_compat(void *rtn)
{
- struct kvm_cpu_compat_check *c = data;
-
- *c->ret = kvm_arch_check_processor_compat(c->opaque);
+ *(int *)rtn = kvm_arch_check_processor_compat();
}
int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
struct module *module)
{
- struct kvm_cpu_compat_check c;
int r;
int cpu;
@@ -5686,10 +5678,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
if (r < 0)
goto out_free_1;
- c.ret = &r;
- c.opaque = opaque;
for_each_online_cpu(cpu) {
- smp_call_function_single(cpu, check_processor_compat, &c, 1);
+ smp_call_function_single(cpu, check_processor_compat, &r, 1);
if (r < 0)
goto out_free_2;
}
--
2.25.1
^ permalink raw reply related
* Re: [PATCH v3 07/10] powerpc/vas: Add paste address mmap fault handler
From: Haren Myneni @ 2022-02-16 2:21 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644808853.k17l68pv15.astroid@bobo.none>
On Mon, 2022-02-14 at 13:37 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:59 am:
> > The user space opens VAS windows and issues NX requests by pasting
> > CRB on the corresponding paste address mmap. When the system looses
>
> s/loose/lose/g throughout the series.
>
> > credits due to core removal, the kernel has to close the window in
> > the hypervisor
>
> By the way what if the kernel does not close the window and we try
> to access the memory? The hypervisor will inject faults?
The requests on the already opened windows will be successful even the
LPAR lost credits (due to core removal). But the hypervisor expects the
LPAR to behave like good citizen and give up resources with core
removal. So we do not see any issue with current upstream code for
DLPAR removal.
But we will have an issue with the migration. The hypervisor knows the
actulal number of credits assigned to the source LPAR before migration.
So assigns the same number on the destination.
>
> > and make the window inactive by unmapping this paste
> > address. Also the OS has to handle NX request page faults if the
> > user
> > space issue NX requests.
> >
> > This handler remap the new paste address with the same VMA when the
> > window is active again (due to core add with DLPAR). Otherwise
> > returns paste failure.
>
> This patch should come before (or combined with) the patch that zaps
> PTEs. Putting it afterwards is logically backward. Even if you don't
> really expect the series to half work in a half bisected state, it
> just makes the changes easier to follow.
>
> Thanks,
> Nick
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/platforms/book3s/vas-api.c | 60
> > +++++++++++++++++++++++++
> > 1 file changed, 60 insertions(+)
> >
> > diff --git a/arch/powerpc/platforms/book3s/vas-api.c
> > b/arch/powerpc/platforms/book3s/vas-api.c
> > index 2d06bd1b1935..5ceba75c13eb 100644
> > --- a/arch/powerpc/platforms/book3s/vas-api.c
> > +++ b/arch/powerpc/platforms/book3s/vas-api.c
> > @@ -351,6 +351,65 @@ static int coproc_release(struct inode *inode,
> > struct file *fp)
> > return 0;
> > }
> >
> > +/*
> > + * This fault handler is invoked when the VAS/NX generates page
> > fault on
> > + * the paste address.
>
> The core generates the page fault here, right? paste destination is
> translated by the core MMU (the instruction is executed in the core,
> afterall).
correct. Will update.
>
> > Happens if the kernel closes window in hypervisor
> > + * (on PowerVM) due to lost credit or the paste address is not
> > mapped.
>
> Call it pseries everywhere if you're talking about the API and Linux
> code, rather than some specific quirk or issue of of the PowerVM
> implementation.
>
> > + */
> > +static vm_fault_t vas_mmap_fault(struct vm_fault *vmf)
> > +{
> > + struct vm_area_struct *vma = vmf->vma;
> > + struct file *fp = vma->vm_file;
> > + struct coproc_instance *cp_inst = fp->private_data;
> > + struct vas_window *txwin;
> > + u64 paste_addr;
> > + int ret;
> > +
> > + /*
> > + * window is not opened. Shouldn't expect this error.
> > + */
> > + if (!cp_inst || !cp_inst->txwin) {
> > + pr_err("%s(): No send window open?\n", __func__);
>
> Probably don't put PR_ERROR logs with question marks in them. The
> administrator knows less than you to answer the question.
>
> "Unexpected fault on paste address with TX window closed" etc.
>
> Then you don't need the comment either because the message explains
> it.
>
> > + return VM_FAULT_SIGBUS;
> > + }
> > +
> > + txwin = cp_inst->txwin;
> > + /*
> > + * Fault is coming due to missing from the original mmap.
>
> Rather than a vague comment like this (which we already know a fault
> comes from a missing or insufficient PTE), you could point to exactly
> the code which zaps the PTEs.
>
> > + * Can happen only when the window is closed due to lost
> > + * credit before mmap() or the user space issued NX request
> > + * without mapping.
> > + */
> > + if (txwin->task_ref.vma != vmf->vma) {
> > + pr_err("%s(): No previous mapping with paste
> > address\n",
> > + __func__);
> > + return VM_FAULT_SIGBUS;
> > + }
> > +
> > + mutex_lock(&txwin->task_ref.mmap_mutex);
> > + /*
> > + * The window may be inactive due to lost credit (Ex: core
> > + * removal with DLPAR). When the window is active again when
> > + * the credit is available, remap with the new paste address.
>
> Remap also typically means mapping the same physical memory at a
> different virtual address. So when you say remap with the new paste
> address, in Linux mm terms that means you're mapping the same window
> at a different virtual address.
>
> But you're faulting a different physical address into the same
> virtual.
Yes, we get different physical address and this handler maps the this
paste address at the existing virtual address. will update this
comment.
>
> > + */
> > + if (txwin->status == VAS_WIN_ACTIVE) {
> > + paste_addr = cp_inst->coproc->vops->paste_addr(txwin);
> > + if (paste_addr) {
> > + ret = vmf_insert_pfn(vma, vma->vm_start,
> > + (paste_addr >> PAGE_SHIFT));
> > + mutex_unlock(&txwin->task_ref.mmap_mutex);
> > + return ret;
> > + }
> > + }
> > + mutex_unlock(&txwin->task_ref.mmap_mutex);
>
> Here a comment about how userspace is supposed to handle the
> window-closed condition might be appropriate.
>
> Thanks,
> Nick
>
> > +
> > + return VM_FAULT_SIGBUS;
> > +
> > +}
> > +static const struct vm_operations_struct vas_vm_ops = {
> > + .fault = vas_mmap_fault,
> > +};
> > +
> > static int coproc_mmap(struct file *fp, struct vm_area_struct
> > *vma)
> > {
> > struct coproc_instance *cp_inst = fp->private_data;
> > @@ -417,6 +476,7 @@ static int coproc_mmap(struct file *fp, struct
> > vm_area_struct *vma)
> > paste_addr, vma->vm_start, rc);
> >
> > txwin->task_ref.vma = vma;
> > + vma->vm_ops = &vas_vm_ops;
> >
> > out:
> > mutex_unlock(&txwin->task_ref.mmap_mutex);
> > --
> > 2.27.0
> >
> >
> >
^ permalink raw reply
* Re: [PATCH v3 04/10] powerpc/pseries/vas: Reopen windows with DLPAR core add
From: Haren Myneni @ 2022-02-16 1:38 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644806607.n8posebbp0.astroid@bobo.none>
On Mon, 2022-02-14 at 13:08 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:56 am:
> > VAS windows can be closed in the hypervisor due to lost credits
> > when the core is removed. If these credits are available later
> > for core add, reopen these windows and set them active. When the
> > kernel sees page fault on the paste address, it creates new mapping
> > on the new paste address. Then the user space can continue to use
> > these windows and send HW compression requests to NX successfully.
>
> Any reason to put this before the close windows patch? It would be
> more logical to put it afterwards AFAIKS.
reconfig_open_windows() is just to reopen and set the status flag when
windows are closed. I thought adding handler first before closing /
unmap helps during git bisect.
I can change.
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/include/asm/vas.h | 16 +++
> > arch/powerpc/platforms/book3s/vas-api.c | 1 +
> > arch/powerpc/platforms/pseries/vas.c | 144
> > ++++++++++++++++++++++++
> > arch/powerpc/platforms/pseries/vas.h | 8 +-
> > 4 files changed, 163 insertions(+), 6 deletions(-)
> >
> > diff --git a/arch/powerpc/include/asm/vas.h
> > b/arch/powerpc/include/asm/vas.h
> > index 57573d9c1e09..f1efe86563cc 100644
> > --- a/arch/powerpc/include/asm/vas.h
> > +++ b/arch/powerpc/include/asm/vas.h
> > @@ -29,6 +29,19 @@
> > #define VAS_THRESH_FIFO_GT_QTR_FULL 2
> > #define VAS_THRESH_FIFO_GT_EIGHTH_FULL 3
> >
> > +/*
> > + * VAS window status
> > + */
> > +#define VAS_WIN_ACTIVE 0x0 /* Used in platform
> > independent */
> > + /* vas mmap() */
> > +/* The hypervisor returns these values */
> > +#define VAS_WIN_CLOSED 0x00000001
> > +#define VAS_WIN_INACTIVE 0x00000002 /* Inactive due to HW
> > failure */
> > +#define VAS_WIN_MOD_IN_PROCESS 0x00000003 /* Process of being
> > modified, */
>
> While you're moving these and adding a comment, it would be good to
> list what hcalls they are relevant to. H_QUERY_VAS_WINDOW (which is
> not
> used anywhere yet?) These are also a 1-byte field, so '0x00', '0x01'
> etc
> would be more appropriate.
Yes, these status bits are assigned by the hypervisor and we are not
using / supporting them right now. I just list them as defined in PAPR
NX HCALLs.
For example: OS can modify the existing window with modify HCALL when
the window is used. During this time, the hypervisor return this status
with window query HCALL.
>
> > + /* deallocated, or quiesced
> > */
> > +/* Linux status bits */
> > +#define VAS_WIN_NO_CRED_CLOSE 0x00000004 /* Window is closed
> > due to */
> > + /* lost credit */
>
> This is mixing a user defined bit field with hcall API value field.
> You
> also AFAIKS as yet don't fill in the hypervisor status anywhere.
>
> I would make this it's own field entirely. A boolean would be nice,
> if
> possible.
Yes, HV status bits are not used here.
In case if the window status is reported thorugh sysfs in future,
thought that it will be simpler to have one status flag.
I can add 'hv_status' for the hypervisor status flag in pseries_vas-
window struct and 'status' for linux in vas_window struct.
We also need one more status for migration. So boolean may not be used.
>
> > /*
> > * Get/Set bit fields
> > */
> > @@ -59,6 +72,8 @@ struct vas_user_win_ref {
> > struct pid *pid; /* PID of owner */
> > struct pid *tgid; /* Thread group ID of owner */
> > struct mm_struct *mm; /* Linux process mm_struct */
> > + struct mutex mmap_mutex; /* protects paste address mmap() */
> > + /* with DLPAR close/open
> > windows */
> > };
> >
> > /*
> > @@ -67,6 +82,7 @@ struct vas_user_win_ref {
> > struct vas_window {
> > u32 winid;
> > u32 wcreds_max; /* Window credits */
> > + u32 status;
> > enum vas_cop_type cop;
> > struct vas_user_win_ref task_ref;
> > char *dbgname;
> > diff --git a/arch/powerpc/platforms/book3s/vas-api.c
> > b/arch/powerpc/platforms/book3s/vas-api.c
> > index 4d82c92ddd52..2b0ced611f32 100644
> > --- a/arch/powerpc/platforms/book3s/vas-api.c
> > +++ b/arch/powerpc/platforms/book3s/vas-api.c
> > @@ -316,6 +316,7 @@ static int coproc_ioc_tx_win_open(struct file
> > *fp, unsigned long arg)
> > return PTR_ERR(txwin);
> > }
> >
> > + mutex_init(&txwin->task_ref.mmap_mutex);
> > cp_inst->txwin = txwin;
> >
> > return 0;
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index 2ef56157634f..d9ff73d7704d 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -501,6 +501,7 @@ static int __init get_vas_capabilities(u8 feat,
> > enum vas_cop_feat_type type,
> > memset(vcaps, 0, sizeof(*vcaps));
> > INIT_LIST_HEAD(&vcaps->list);
> >
> > + vcaps->feat = feat;
> > caps = &vcaps->caps;
> >
> > rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, feat,
> > @@ -539,6 +540,145 @@ static int __init get_vas_capabilities(u8
> > feat, enum vas_cop_feat_type type,
> > return 0;
> > }
> >
> > +/*
> > + * VAS windows can be closed due to lost credits when the core is
> > + * removed. So reopen them if credits are available due to DLPAR
> > + * core add and set the window active status. When NX sees the
> > page
> > + * fault on the unmapped paste address, the kernel handles the
> > fault
> > + * by setting the remapping to new paste address if the window is
> > + * active.
> > + */
> > +static int reconfig_open_windows(struct vas_caps *vcaps, int
> > creds)
> > +{
> > + long domain[PLPAR_HCALL9_BUFSIZE] = {VAS_DEFAULT_DOMAIN_ID};
> > + struct vas_cop_feat_caps *caps = &vcaps->caps;
> > + struct pseries_vas_window *win = NULL, *tmp;
> > + int rc, mv_ents = 0;
> > +
> > + /*
> > + * Nothing to do if there are no closed windows.
> > + */
> > + if (!vcaps->close_wins)
> > + return 0;
> > +
> > + /*
> > + * For the core removal, the hypervisor reduces the credits
> > + * assigned to the LPAR and the kernel closes VAS windows
> > + * in the hypervisor depends on reduced credits. The kernel
> > + * uses LIFO (the last windows that are opened will be closed
> > + * first) and expects to open in the same order when credits
> > + * are available.
> > + * For example, 40 windows are closed when the LPAR lost 2
> > cores
> > + * (dedicated). If 1 core is added, this LPAR can have 20 more
> > + * credits. It means the kernel can reopen 20 windows. So move
> > + * 20 entries in the VAS windows lost and reopen next 20
> > windows.
> > + */
> > + if (vcaps->close_wins > creds)
> > + mv_ents = vcaps->close_wins - creds;
> > +
> > + list_for_each_entry_safe(win, tmp, &vcaps->list, win_list) {
> > + if (!mv_ents)
> > + break;
> > +
> > + mv_ents--;
> > + }
> > +
> > + list_for_each_entry_safe_from(win, tmp, &vcaps->list, win_list)
> > {
> > + /*
> > + * Nothing to do on this window if it is not closed
> > + * with VAS_WIN_NO_CRED_CLOSE
> > + */
> > + if (!(win->vas_win.status & VAS_WIN_NO_CRED_CLOSE))
> > + continue;
> > +
> > + rc = allocate_setup_window(win, (u64 *)&domain[0],
> > + caps->win_type);
> > + if (rc)
> > + return rc;
> > +
> > + rc = h_modify_vas_window(win);
> > + if (rc)
> > + goto out;
> > +
> > + mutex_lock(&win->vas_win.task_ref.mmap_mutex);
> > + /*
> > + * Set window status to active
> > + */
> > + win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE;
> > + mutex_unlock(&win->vas_win.task_ref.mmap_mutex);
>
> What's the mutex protecting? If status can change, what happens if
> it
> changed between checking it above and this point?
We need this mutex to protect status flag between
reconfig_close/open_windows() and VAS mmap handler.
We hold pseries_mutex() before calling recnfig_close/popen_windows(),
so not an issue here.
>
> > + win->win_type = caps->win_type;
> > + if (!--vcaps->close_wins)
> > + break;
> > + }
> > +
> > + return 0;
> > +out:
> > + /*
> > + * Window modify HCALL failed. So close the window to the
> > + * hypervisor and return.
> > + */
> > + free_irq_setup(win);
> > + h_deallocate_vas_window(win->vas_win.winid);
> > + return rc;
> > +}
> > +
> > +/*
> > + * Get new VAS capabilities when the core add/removal
> > configuration
> > + * changes. Reconfig window configurations based on the credits
> > + * availability from this new capabilities.
> > + */
> > +static int vas_reconfig_capabilties(u8 type)
> > +{
> > + struct hv_vas_cop_feat_caps *hv_caps;
> > + struct vas_cop_feat_caps *caps;
> > + int lpar_creds, new_creds;
> > + struct vas_caps *vcaps;
> > + int rc = 0;
> > +
> > + if (type >= VAS_MAX_FEAT_TYPE) {
> > + pr_err("Invalid credit type %d\n", type);
> > + return -EINVAL;
> > + }
> > +
> > + vcaps = &vascaps[type];
> > + caps = &vcaps->caps;
> > +
> > + hv_caps = kmalloc(sizeof(*hv_caps), GFP_KERNEL);
> > + if (!hv_caps)
> > + return -ENOMEM;
> > +
> > + mutex_lock(&vas_pseries_mutex);
> > + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, vcaps-
> > >feat,
> > + (u64)virt_to_phys(hv_caps));
> > + if (rc)
> > + goto out;
> > +
> > + new_creds = be16_to_cpu(hv_caps->target_lpar_creds);
> > +
> > + lpar_creds = atomic_read(&caps->target_creds);
>
> NBD but it would be slightly clearer that new_creds is the absolute
> value not the number of new credits if you call these two
> new_nr_creds and old_nr_creds or prev_nr_creds.
Sure will change.
>
> > +
> > + atomic_set(&caps->target_creds, new_creds);
> > + /*
> > + * The total number of available credits may be decreased or
> > + * inceased with DLPAR operation. Means some windows have to be
> > + * closed / reopened. Hold the vas_pseries_mutex so that the
> > + * the user space can not open new windows.
> > + */
> > + if (lpar_creds < new_creds) {
> > + /*
> > + * If the existing target credits is less than the new
> > + * target, reopen windows if they are closed due to
> > + * the previous DLPAR (core removal).
> > + */
> > + rc = reconfig_open_windows(vcaps, new_creds -
> > lpar_creds);
> > + }
> > +
> > +out:
> > + mutex_unlock(&vas_pseries_mutex);
> > + kfree(hv_caps);
> > + return rc;
> > +}
> > +
> > /*
> > * Total number of default credits available (target_credits)
> > * in LPAR depends on number of cores configured. It varies based
> > on
> > @@ -565,6 +705,10 @@ static int pseries_vas_notifier(struct
> > notifier_block *nb,
> > if (!intserv)
> > return NOTIFY_OK;
> >
> > + rc = vas_reconfig_capabilties(VAS_GZIP_DEF_FEAT_TYPE);
> > + if (rc)
> > + pr_err("Failed reconfig VAS capabilities with
> > DLPAR\n");
> > +
> > return rc;
> > }
> >
> > diff --git a/arch/powerpc/platforms/pseries/vas.h
> > b/arch/powerpc/platforms/pseries/vas.h
> > index 0538760d13be..45b62565955b 100644
> > --- a/arch/powerpc/platforms/pseries/vas.h
> > +++ b/arch/powerpc/platforms/pseries/vas.h
> > @@ -21,12 +21,6 @@
> > #define VAS_MOD_WIN_FLAGS (VAS_MOD_WIN_JOBS_KILL | VAS_MOD_WIN_DR
> > | \
> > VAS_MOD_WIN_PR | VAS_MOD_WIN_SF)
> >
> > -#define VAS_WIN_ACTIVE 0x0
> > -#define VAS_WIN_CLOSED 0x1
> > -#define VAS_WIN_INACTIVE 0x2 /* Inactive due to HW failure */
> > -/* Process of being modified, deallocated, or quiesced */
> > -#define VAS_WIN_MOD_IN_PROCESS 0x3
> > -
> > #define VAS_COPY_PASTE_USER_MODE 0x00000001
> > #define VAS_COP_OP_USER_MODE 0x00000010
> >
> > @@ -84,6 +78,8 @@ struct vas_cop_feat_caps {
> > struct vas_caps {
> > struct vas_cop_feat_caps caps;
> > struct list_head list; /* List of open windows */
> > + int close_wins; /* closed windows in the hypervisor
> > for DLPAR */
>
> 'nr_closed_wins'
>
> This does not look like a capability. Although I guess neither do
> some
> of the fields (used_creds) in vas_cop_feat_caps. Why not at least put
> this there with the rest of those credits fields?
We can add used_credits() here since it is not part of HV capability.
It was included in previous pathes, So I did not change,
Thanks
Haren
>
> Thanks,
> Nick
>
> > + u8 feat; /* Feature type */
> > };
> >
> > /*
> > --
> > 2.27.0
> >
> >
> >
^ permalink raw reply
* Re: [PATCH v3 06/10] powerpc/vas: Map paste address only if window is active
From: Haren Myneni @ 2022-02-16 1:58 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644808630.pfiol54rgg.astroid@bobo.none>
On Mon, 2022-02-14 at 13:20 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:58 am:
> > The paste address mapping is done with mmap() after the window is
> > opened with ioctl. But the window can be closed due to lost credit
> > due to core removal before mmap(). So if the window is not active,
> > return mmap() failure with -EACCES and expects the user space
> > reissue
> > mmap() when the window is active or open new window when the credit
> > is available.
> >
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/platforms/book3s/vas-api.c | 21 ++++++++++++++++++++-
> > 1 file changed, 20 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/platforms/book3s/vas-api.c
> > b/arch/powerpc/platforms/book3s/vas-api.c
> > index a63fd48e34a7..2d06bd1b1935 100644
> > --- a/arch/powerpc/platforms/book3s/vas-api.c
> > +++ b/arch/powerpc/platforms/book3s/vas-api.c
> > @@ -379,10 +379,27 @@ static int coproc_mmap(struct file *fp,
> > struct vm_area_struct *vma)
> > return -EACCES;
> > }
> >
> > + /*
> > + * The initial mapping is done after the window is opened
> > + * with ioctl. But this window might have been closed
> > + * due to lost credit (core removal on PowerVM) before mmap().
>
> What does "initial mapping" mean?
>
> mapping ~= mmap, in kernel speak.
yes, the initial mapping is done with the actual mmap() call.
>
> You will have to differentiate the concepts.
>
> > + * So if the window is not active, return mmap() failure
> > + * with -EACCES and expects the user space reconfigure (mmap)
> > + * window when it is active again or open new window when
> > + * the credit is available.
> > + */
> > + mutex_lock(&txwin->task_ref.mmap_mutex);
> > + if (txwin->status != VAS_WIN_ACTIVE) {
> > + pr_err("%s(): Window is not active\n", __func__);
> > + rc = -EACCES;
> > + goto out;
> > + }
> > +
> > paste_addr = cp_inst->coproc->vops->paste_addr(txwin);
> > if (!paste_addr) {
> > pr_err("%s(): Window paste address failed\n",
> > __func__);
> > - return -EINVAL;
> > + rc = -EINVAL;
> > + goto out;
> > }
> >
> > pfn = paste_addr >> PAGE_SHIFT;
> > @@ -401,6 +418,8 @@ static int coproc_mmap(struct file *fp, struct
> > vm_area_struct *vma)
> >
> > txwin->task_ref.vma = vma;
> >
> > +out:
> > + mutex_unlock(&txwin->task_ref.mmap_mutex);
>
> If the hypervisor can revoke a window at any point with DLPAR, it's
> not
> clear *why* this is needed. The hypervisor could cause your window
> to
> close right after this mmap() returns, right? So an explanation for
> exactly what this patch is needed for beyond that would help.
Yes, the window can be closed by OS due to DLPAR after the mmap()
returns successfully which is a normal case - paste instruction failure
until the window is reopened again.
But ths patch is mainly for window open by user space and dlpar happens
before the user space issue mmap().
I will add more description in the commit log.
Thanks
Haren
>
> Thanks,
> Nick
^ permalink raw reply
* Re: [PATCH v3 05/10] powerpc/pseries/vas: Close windows with DLPAR core removal
From: Haren Myneni @ 2022-02-16 1:48 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644806848.01buhj7y5b.astroid@bobo.none>
On Mon, 2022-02-14 at 13:17 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:57 am:
> > The hypervisor reduces the available credits if the core is removed
> > from the LPAR. So there is possibility of using excessive credits
> > (windows) in the LPAR and the hypervisor expects the system to
> > close
> > the excessive windows. Even though the user space can continue to
> > use
> > these windows to send compression requests to NX, the hypervisor
> > expects
> > the LPAR to reduce these windows usage so that NX load can be
> > equally
> > distributed across all LPARs in the system.
> >
> > When the DLPAR notifier is received, get the new VAS capabilities
> > from
> > the hypervisor and close the excessive windows in the hypervisor.
> > Also
> > the kernel unmaps the paste address so that the user space receives
> > paste
> > failure until these windows are active with the later DLPAR (core
> > add).
>
> The changelog needs work. Unmapping the window and the ramifications
> of
> that needs more description here or in comments.
Thanks will change.
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/include/asm/vas.h | 1 +
> > arch/powerpc/platforms/book3s/vas-api.c | 2 +
> > arch/powerpc/platforms/pseries/vas.c | 117
> > ++++++++++++++++++++++--
> > arch/powerpc/platforms/pseries/vas.h | 1 +
> > 4 files changed, 112 insertions(+), 9 deletions(-)
> >
> > diff --git a/arch/powerpc/include/asm/vas.h
> > b/arch/powerpc/include/asm/vas.h
> > index f1efe86563cc..ddc05a8fc2e3 100644
> > --- a/arch/powerpc/include/asm/vas.h
> > +++ b/arch/powerpc/include/asm/vas.h
> > @@ -74,6 +74,7 @@ struct vas_user_win_ref {
> > struct mm_struct *mm; /* Linux process mm_struct */
> > struct mutex mmap_mutex; /* protects paste address mmap() */
> > /* with DLPAR close/open
> > windows */
> > + struct vm_area_struct *vma; /* Save VMA and used in DLPAR ops
> > */
> > };
> >
> > /*
> > diff --git a/arch/powerpc/platforms/book3s/vas-api.c
> > b/arch/powerpc/platforms/book3s/vas-api.c
> > index 2b0ced611f32..a63fd48e34a7 100644
> > --- a/arch/powerpc/platforms/book3s/vas-api.c
> > +++ b/arch/powerpc/platforms/book3s/vas-api.c
> > @@ -399,6 +399,8 @@ static int coproc_mmap(struct file *fp, struct
> > vm_area_struct *vma)
> > pr_devel("%s(): paste addr %llx at %lx, rc %d\n", __func__,
> > paste_addr, vma->vm_start, rc);
> >
> > + txwin->task_ref.vma = vma;
> > +
> > return rc;
> > }
> >
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index d9ff73d7704d..75ccd0a599ec 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -370,13 +370,28 @@ static struct vas_window
> > *vas_allocate_window(int vas_id, u64 flags,
> > if (rc)
> > goto out_free;
> >
> > - vas_user_win_add_mm_context(&txwin->vas_win.task_ref);
> > txwin->win_type = cop_feat_caps->win_type;
> > mutex_lock(&vas_pseries_mutex);
> > - list_add(&txwin->win_list, &caps->list);
> > + /*
> > + * Possible to loose the acquired credit with DLPAR core
>
> s/loose/lose/g
>
> > + * removal after the window is opened. So if there are any
> > + * closed windows (means with lost credits), do not give new
> > + * window to user space. New windows will be opened only
> > + * after the existing windows are reopened when credits are
> > + * available.
> > + */
> > + if (!caps->close_wins) {
> > + list_add(&txwin->win_list, &caps->list);
> > + caps->num_wins++;
> > + mutex_unlock(&vas_pseries_mutex);
> > + vas_user_win_add_mm_context(&txwin->vas_win.task_ref);
> > + return &txwin->vas_win;
> > + }
> > mutex_unlock(&vas_pseries_mutex);
> >
> > - return &txwin->vas_win;
> > + put_vas_user_win_ref(&txwin->vas_win.task_ref);
> > + rc = -EBUSY;
> > + pr_err("No credit is available to allocate window\n");
> >
> > out_free:
> > /*
> > @@ -439,14 +454,24 @@ static int vas_deallocate_window(struct
> > vas_window *vwin)
> >
> > caps = &vascaps[win->win_type].caps;
> > mutex_lock(&vas_pseries_mutex);
> > - rc = deallocate_free_window(win);
> > - if (rc) {
> > - mutex_unlock(&vas_pseries_mutex);
> > - return rc;
> > - }
> > + /*
> > + * VAS window is already closed in the hypervisor when
> > + * lost the credit. So just remove the entry from
> > + * the list, remove task references and free vas_window
> > + * struct.
> > + */
> > + if (win->vas_win.status & VAS_WIN_NO_CRED_CLOSE) {
> > + rc = deallocate_free_window(win);
> > + if (rc) {
> > + mutex_unlock(&vas_pseries_mutex);
> > + return rc;
> > + }
> > + } else
> > + vascaps[win->win_type].close_wins--;
> >
> > list_del(&win->win_list);
> > atomic_dec(&caps->used_creds);
> > + vascaps[win->win_type].num_wins--;
> > mutex_unlock(&vas_pseries_mutex);
> >
> > put_vas_user_win_ref(&vwin->task_ref);
> > @@ -622,6 +647,72 @@ static int reconfig_open_windows(struct
> > vas_caps *vcaps, int creds)
> > return rc;
> > }
> >
> > +/*
> > + * The hypervisor reduces the available credits if the LPAR lost
> > core. It
> > + * means the excessive windows should not be active and the user
> > space
> > + * should not be using these windows to send compression requests
> > to NX.
> > + * So the kernel closes the excessive windows and unmap the paste
> > address
> > + * such that the user space receives paste instruction failure.
> > Then up to
> > + * the user space to fall back to SW compression and manage with
> > the
> > + * existing windows.
> > + */
> > +static int reconfig_close_windows(struct vas_caps *vcap, int
> > excess_creds)
> > +{
> > + struct pseries_vas_window *win, *tmp;
> > + struct vas_user_win_ref *task_ref;
> > + struct vm_area_struct *vma;
> > + int rc = 0;
> > +
> > + list_for_each_entry_safe(win, tmp, &vcap->list, win_list) {
> > + /*
> > + * This window is already closed due to lost credit
> > + * before. Go for next window.
> > + */
> > + if (win->vas_win.status & VAS_WIN_NO_CRED_CLOSE)
> > + continue;
> > +
> > + task_ref = &win->vas_win.task_ref;
> > + mutex_lock(&task_ref->mmap_mutex);
> > + vma = task_ref->vma;
> > + /*
> > + * Number of available credits are reduced, So select
> > + * and close windows.
> > + */
> > + win->vas_win.status |= VAS_WIN_NO_CRED_CLOSE;
> > +
> > + mmap_write_lock(task_ref->mm);
> > + /*
> > + * vma is set in the original mapping. But this mapping
> > + * is done with mmap() after the window is opened with
> > ioctl.
> > + * so we may not see the original mapping if the core
> > remove
> > + * is done before the original mmap() and after the
> > ioctl.
> > + */
> > + if (vma)
> > + zap_page_range(vma, vma->vm_start,
> > + vma->vm_end - vma->vm_start);
> > +
> > + mmap_write_unlock(task_ref->mm);
> > + mutex_unlock(&task_ref->mmap_mutex);
> > + /*
> > + * Close VAS window in the hypervisor, but do not
> > + * free vas_window struct since it may be reused
> > + * when the credit is available later (DLPAR with
> > + * adding cores). This struct will be used
> > + * later when the process issued with close(FD).
> > + */
> > + rc = deallocate_free_window(win);
> > + if (rc)
> > + return rc;
> > +
> > + vcap->close_wins++;
> > +
> > + if (!--excess_creds)
> > + break;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > /*
> > * Get new VAS capabilities when the core add/removal
> > configuration
> > * changes. Reconfig window configurations based on the credits
> > @@ -633,7 +724,7 @@ static int vas_reconfig_capabilties(u8 type)
> > struct vas_cop_feat_caps *caps;
> > int lpar_creds, new_creds;
> > struct vas_caps *vcaps;
> > - int rc = 0;
> > + int rc = 0, active_wins;
> >
> > if (type >= VAS_MAX_FEAT_TYPE) {
> > pr_err("Invalid credit type %d\n", type);
> > @@ -671,6 +762,14 @@ static int vas_reconfig_capabilties(u8 type)
> > * the previous DLPAR (core removal).
> > */
> > rc = reconfig_open_windows(vcaps, new_creds -
> > lpar_creds);
> > + } else {
> > + /*
> > + * # active windows is more than new LPAR available
> > + * credits. So close the excessive windows.
> > + */
> > + active_wins = vcaps->num_wins - vcaps->close_wins;
> > + if (active_wins > new_creds)
> > + rc = reconfig_close_windows(vcaps, active_wins
> > - new_creds);
> > }
> >
> > out:
> > diff --git a/arch/powerpc/platforms/pseries/vas.h
> > b/arch/powerpc/platforms/pseries/vas.h
> > index 45b62565955b..8ce9b84693e8 100644
> > --- a/arch/powerpc/platforms/pseries/vas.h
> > +++ b/arch/powerpc/platforms/pseries/vas.h
> > @@ -79,6 +79,7 @@ struct vas_caps {
> > struct vas_cop_feat_caps caps;
> > struct list_head list; /* List of open windows */
> > int close_wins; /* closed windows in the hypervisor
> > for DLPAR */
> > + int num_wins; /* Number of windows opened */
>
> num -> nr is usually best.
>
> And I know you like to abbreviate names, but it helps unfamiliar
> readers
> to use a few more letters. num_wins -> nr_windows or even better
> nr_open_windows.
>
> Same comment about location of the field. Also it's not quite clear
> to
> me what the relationship is between your open windows here and used
> credits in the vas_cop_feat_caps structure. In other comments and
> changelogs you use credit and window interchangeably.
will change to nr_close_windows and nr_windows.
used_credits is used for how many windows / credits are used by user
space. But when OS lost the credit, this window will be used in the
hypervisor, but not from the user space point of view. It should expect
this window is not active and should be available later.
But OS will be closing this window in the hypervisor. So used
close_wins / num_wins are used to track the actual number of windows
opened/closed in the hypervisor.
Thanks
Haren
>
>
> > u8 feat; /* Feature type */
> > };
> >
> > --
> > 2.27.0
> >
> >
> >
^ permalink raw reply
* Re: [PATCH kernel 2/3] powerpc/llvm: Sample config for LLVM LTO
From: Alexey Kardashevskiy @ 2022-02-16 1:28 UTC (permalink / raw)
To: Nick Desaulniers
Cc: Nathan Lynch, Fabiano Rosas, llvm, Nicholas Piggin,
Nathan Chancellor, Joel Stanley, Naveen N. Rao, linuxppc-dev,
Daniel Axtens
In-Reply-To: <CAKwvOdmJspxuNh-K-CeLJh4X4GGVYJWH29HnjJwuRL+jcN9G5w@mail.gmail.com>
On 2/12/22 11:05, Nick Desaulniers wrote:
> On Thu, Feb 10, 2022 at 6:31 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>>
>> The config is a copy of ppc64_defconfig with a few tweaks. This could be
>> a smaller config to merge into ppc64_defconfig but unfortunately
>> merger does not allow disabling already enabled options.
>
> Cool series!
>
>>
>> This is a command line to compile the kernel using the upstream llvm:
>>
>> make -j64 O=/home/aik/pbuild/kernels-llvm/ \
>> "KCFLAGS=-Wmissing-braces -Wno-array-bounds" \
>> ARCH=powerpc LLVM_IAS=1 ppc64le_lto_defconfig CC=clang LLVM=1
>
> That command line invocation is kind of a mess, and many things
> shouldn't be necessary.
>
> O= is just noise; if folks are doing in tree builds then that doesn't
> add anything meaningful.
> KCFLAGS= why? I know -Warray-bounds is being worked on actively, but
> do we have instances of -Wmissing-braces at the moment? Let's get
> those fixed up.
> LLVM_IAS=1 is implied by LLVM=1.
> CC=clang is implied by LLVM=1
>
> why add a new config? I think it would be simpler to just show command
> line invocations of `./scripts/config -e` and `make`. No new config
> required.
I should have added "RFC" in this one as the purpose of the patch is to
show what works right now and not for actual submission.
>>
>> Forces CONFIG_BTRFS_FS=y to make CONFIG_ZSTD_COMPRESS=y to fix:
>> ld.lld: error: linking module flags 'Code Model': IDs have conflicting values in 'lib/built-in.a(entropy_common.o at 5332)' and 'ld-temp.o'
>>
>> because modules are linked with -mcmodel=large but the kernel uses -mcmodel=medium
>
> Please file a bug about this.
> https://github.com/ClangBuiltLinux/linux/issues
>
>>
>> Enables CONFIG_USERFAULTFD=y as otherwise vm_userfaultfd_ctx becomes
>> 0 bytes long and clang sanitizer crashes as
>> https://bugs.llvm.org/show_bug.cgi?id=500375
>
> The above hyperlink doesn't work for me. Upstream llvm just moved from
> bugzilla to github issue tracker.
aah this is the correct one:
https://bugs.llvm.org/show_bug.cgi?id=50037
>
> https://github.com/llvm/llvm-project/issues
oh ok.
>>
>> Disables CONFIG_FTR_FIXUP_SELFTEST as it uses FTR_SECTION_ELSE with
>> conditional branches. There are other places like this and the following
>> patches address that.
>>
>> Disables CONFIG_FTRACE_MCOUNT_USE_RECORDMCOUNT as CONFIG_HAS_LTO_CLANG
>> depends on it being disabled. In order to avoid disabling way too many
>> options (like DYNAMIC_FTRACE/FUNCTION_TRACER), this converts
>> FTRACE_MCOUNT_USE_RECORDMCOUNT from def_bool to bool.
>>
>> Note that even with this config there is a good chance that LTO
>> is going to fail linking vmlinux because of the "bc" problem.
>
> I think rather than adding a new config with LTO enabled and a few
> things turned off, it would be better to not allow LTO to be
> selectable if those things are turned on, until the combination of the
> two are fixed.
Well, if I want people to try this thing, I kinda need to provide an
easy way to allow LTO. The new config seemed the easiest (== the
shortest) :)
^ permalink raw reply
* Re: [PATCH v3 03/10] powerpc/pseries/vas: Save LPID in pseries_vas_window struct
From: Haren Myneni @ 2022-02-16 1:09 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644805860.xhvkad4duc.astroid@bobo.none>
On Mon, 2022-02-14 at 12:41 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:55 am:
> > The kernel sets the VAS window with partition PID when is opened in
> > the hypervisor. During DLPAR operation, windows can be closed and
> > reopened in the hypervisor when the credit is available. So saves
> > this PID in pseries_vas_window struct when the window is opened
> > initially and reuse it later during DLPAR operation.
>
> This probably shouldn't be called lpid, while you're changing it.
> "partition PID" and "LPAR PID" is also confusing. I know the name
> somewhat comes from the specifiction, but pid/PID would be fine,
> it's clear we are talking about "this LPAR" when in pseries code.
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/platforms/pseries/vas.c | 7 ++++---
> > arch/powerpc/platforms/pseries/vas.h | 1 +
> > 2 files changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index d2c8292bfb33..2ef56157634f 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -107,7 +107,6 @@ static int h_deallocate_vas_window(u64 winid)
> > static int h_modify_vas_window(struct pseries_vas_window *win)
> > {
> > long rc;
> > - u32 lpid = mfspr(SPRN_PID);
> >
> > /*
> > * AMR value is not supported in Linux VAS implementation.
> > @@ -115,7 +114,7 @@ static int h_modify_vas_window(struct
> > pseries_vas_window *win)
> > */
> > do {
> > rc = plpar_hcall_norets(H_MODIFY_VAS_WINDOW,
> > - win->vas_win.winid, lpid, 0,
> > + win->vas_win.winid, win->lpid,
> > 0,
> > VAS_MOD_WIN_FLAGS, 0);
> >
> > rc = hcall_return_busy_check(rc);
> > @@ -125,7 +124,7 @@ static int h_modify_vas_window(struct
> > pseries_vas_window *win)
> > return 0;
> >
> > pr_err("H_MODIFY_VAS_WINDOW error: %ld, winid %u lpid %u\n",
> > - rc, win->vas_win.winid, lpid);
> > + rc, win->vas_win.winid, win->lpid);
> > return -EIO;
> > }
> >
> > @@ -338,6 +337,8 @@ static struct vas_window
> > *vas_allocate_window(int
> > vas_id, u64 flags,
> > }
> > }
> >
> > + txwin->lpid = mfspr(SPRN_PID);
> > +
> > /*
> > * Allocate / Deallocate window hcalls and setup / free IRQs
> > * have to be protected with mutex.
> > diff --git a/arch/powerpc/platforms/pseries/vas.h
> > b/arch/powerpc/platforms/pseries/vas.h
> > index fa7ce74f1e49..0538760d13be 100644
> > --- a/arch/powerpc/platforms/pseries/vas.h
> > +++ b/arch/powerpc/platforms/pseries/vas.h
> > @@ -115,6 +115,7 @@ struct pseries_vas_window {
> > u64 domain[6]; /* Associativity domain Ids */
> > /* this window is allocated */
> > u64 util;
> > + u32 lpid;
>
> Comment could be "PID associated with this window".
yes, will add this comment.
>
> BTW, is the TID parameter deprecated? Doesn't seem that we use that.
Right, tpid is deprecated on p10 and we are not using it.
Thanks
Haren
>
> Thanks,
> Nick
^ permalink raw reply
* Re: [PATCH v3 02/10] powerpc/pseries/vas: Add notifier for DLPAR core removal/add
From: Haren Myneni @ 2022-02-16 1:07 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe; +Cc: Nathan Lynch
In-Reply-To: <1644804881.mp6yrf6sdz.astroid@bobo.none>
On Mon, 2022-02-14 at 12:27 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:54 am:
> > The hypervisor assigns credits for each LPAR based on number of
> > cores configured in that system. So expects to release credits
> > (means windows) when the core is removed. This patch adds notifier
> > for core removal/add so that the OS closes windows if the system
> > looses credits due to core removal and reopen windows when the
> > credits available later.
>
> This could be improved. As far as I can tell,
>
> The hypervisor assigns vas credits (windows) for each LPAR based on
> the
> number of cores configured in that system. The OS is expected to
> release credits when cores are removed, and may allocate more when
> cores are added.
>
> Or can you only re-use credits that you previously lost?
yes, reopen windows / re-use credits when the previously lost credits
are available.
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/platforms/pseries/vas.c | 37
> > ++++++++++++++++++++++++++++
> > 1 file changed, 37 insertions(+)
> >
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index c0737379cc7b..d2c8292bfb33 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -538,6 +538,39 @@ static int __init get_vas_capabilities(u8
> > feat, enum vas_cop_feat_type type,
> > return 0;
> > }
> >
> > +/*
> > + * Total number of default credits available (target_credits)
> > + * in LPAR depends on number of cores configured. It varies based
> > on
> > + * whether processors are in shared mode or dedicated mode.
> > + * Get the notifier when CPU configuration is changed with DLPAR
> > + * operation so that get the new target_credits (vas default
> > capabilities)
> > + * and then update the existing windows usage if needed.
> > + */
> > +static int pseries_vas_notifier(struct notifier_block *nb,
> > + unsigned long action, void *data)
> > +{
> > + struct of_reconfig_data *rd = data;
> > + struct device_node *dn = rd->dn;
> > + const __be32 *intserv = NULL;
> > + int len, rc = 0;
> > +
> > + if ((action == OF_RECONFIG_ATTACH_NODE) ||
> > + (action == OF_RECONFIG_DETACH_NODE))
>
> I suppose the OF notifier is the way to do it (cc Nathan).
Using notifier here. registering notifier
with of_reconfig_notifier_register() as in other places (hotplug-
cpu.c pseries_smp_notifier())
>
> Could this patch be folded in with where it acually does something?
> It
> makes it easier to review and understand how the notifier is used.
Added this notifier as a seperate patch to make it smaller. Sure, I can
include this patch in 'Add reconfig_close/open_windows() patch'.
>
>
> > + intserv = of_get_property(dn, "ibm,ppc-interrupt-
> > server#s",
> > + &len);
> > + /*
> > + * Processor config is not changed
> > + */
> > + if (!intserv)
> > + return NOTIFY_OK;
> > +
> > + return rc;
> > +}
> > +
> > +static struct notifier_block pseries_vas_nb = {
> > + .notifier_call = pseries_vas_notifier,
> > +};
> > +
> > static int __init pseries_vas_init(void)
> > {
> > struct hv_vas_cop_feat_caps *hv_cop_caps;
> > @@ -591,6 +624,10 @@ static int __init pseries_vas_init(void)
> > goto out_cop;
> > }
> >
> > + /* Processors can be added/removed only on LPAR */
>
> What does this comment mean? DLPAR?
I will remve it, basically trying to say that this notifier is called
when core is removed / added.
Thanks
haren
>
> Thanks,
> Nick
>
> > + if (copypaste_feat && firmware_has_feature(FW_FEATURE_LPAR))
> > + of_reconfig_notifier_register(&pseries_vas_nb);
> > +
> > pr_info("GZIP feature is available\n");
> >
> > out_cop:
> > --
> > 2.27.0
> >
> >
> >
^ permalink raw reply
* Re: [PATCH v1 1/2] cma: factor out minimum alignment requirement
From: Rob Herring @ 2022-02-15 19:53 UTC (permalink / raw)
To: David Hildenbrand
Cc: devicetree, Michael S. Tsirkin, Aneesh Kumar K V, linuxppc-dev,
linux-kernel@vger.kernel.org,
open list:DRM DRIVER FOR QEMU'S CIRRUS DEVICE, linux-mm,
Minchan Kim, Linux IOMMU, Paul Mackerras, Zi Yan, Robin Murphy,
Andrew Morton, Frank Rowand, Christoph Hellwig, Vlastimil Babka,
Marek Szyprowski
In-Reply-To: <20220214174132.219303-2-david@redhat.com>
On Mon, Feb 14, 2022 at 11:42 AM David Hildenbrand <david@redhat.com> wrote:
>
> Let's factor out determining the minimum alignment requirement for CMA
> and add a helpful comment.
>
> No functional change intended.
>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
> arch/powerpc/include/asm/fadump-internal.h | 5 -----
> arch/powerpc/kernel/fadump.c | 2 +-
> drivers/of/of_reserved_mem.c | 9 +++------
Acked-by: Rob Herring <robh@kernel.org>
> include/linux/cma.h | 9 +++++++++
> kernel/dma/contiguous.c | 4 +---
> mm/cma.c | 20 +++++---------------
> 6 files changed, 19 insertions(+), 30 deletions(-)
^ permalink raw reply
* [PATCH v2 1/2] crypto: vmx: Turn CRYPTO_DEV_VMX_ENCRYPT into tristate
From: Petr Vorel @ 2022-02-15 18:59 UTC (permalink / raw)
To: linux-crypto
Cc: Herbert Xu, Nicolai Stange, linux-kbuild, Nayna Jain, Petr Vorel,
Paulo Flabiano Smorigo, leitao, linuxppc-dev
In-Reply-To: <20220215185936.15576-1-pvorel@suse.cz>
and remove CRYPTO_DEV_VMX, which looked redundant when only
CRYPTO_DEV_VMX_ENCRYPT used it. Also it forces CRYPTO_GHASH to be
builtin even CRYPTO_DEV_VMX_ENCRYPT was configured as module.
Update powerpc defconfigs and description in MAINTAINERS.
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
new in v2
This might be a bit aggressive, but IMHO CRYPTO_DEV_VMX only complicated
things for nothing. But if you do *not* agree with removing it, I just add
select to drivers/crypto/vmx/Kconfig (which forces dependencies to be
always modules.)
If it's ok for you to remove, please also check whether the description
is ok. get_maintainer.pl script has size limitation:
$ ./scripts/get_maintainer.pl drivers/crypto/vmx/Kconfig
...
linux-crypto@vger.kernel.org (open list:IBM Power VMX Cryptographic Acceleration Instru...)
maybe the name should be shorter.
Kind regards,
Petr
MAINTAINERS | 2 +-
arch/powerpc/configs/powernv_defconfig | 2 +-
arch/powerpc/configs/ppc64_defconfig | 2 +-
arch/powerpc/configs/pseries_defconfig | 2 +-
drivers/crypto/Kconfig | 6 ------
drivers/crypto/vmx/Kconfig | 4 ++--
6 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index ea3e6c914384..80e562579180 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9207,7 +9207,7 @@ L: target-devel@vger.kernel.org
S: Supported
F: drivers/scsi/ibmvscsi_tgt/
-IBM Power VMX Cryptographic instructions
+IBM Power VMX Cryptographic Acceleration Instructions Driver
M: Breno Leitão <leitao@debian.org>
M: Nayna Jain <nayna@linux.ibm.com>
M: Paulo Flabiano Smorigo <pfsmorigo@gmail.com>
diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig
index 49f49c263935..4b250d05dcdf 100644
--- a/arch/powerpc/configs/powernv_defconfig
+++ b/arch/powerpc/configs/powernv_defconfig
@@ -337,7 +337,7 @@ CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_DEV_NX=y
-CONFIG_CRYPTO_DEV_VMX=y
+CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m
CONFIG_VIRTUALIZATION=y
CONFIG_KVM_BOOK3S_64=m
CONFIG_KVM_BOOK3S_64_HV=m
diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig
index c8b0e80d613b..ebd33b94debb 100644
--- a/arch/powerpc/configs/ppc64_defconfig
+++ b/arch/powerpc/configs/ppc64_defconfig
@@ -355,7 +355,7 @@ CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_DEV_NX=y
CONFIG_CRYPTO_DEV_NX_ENCRYPT=m
-CONFIG_CRYPTO_DEV_VMX=y
+CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m
CONFIG_PRINTK_TIME=y
CONFIG_PRINTK_CALLER=y
CONFIG_MAGIC_SYSRQ=y
diff --git a/arch/powerpc/configs/pseries_defconfig b/arch/powerpc/configs/pseries_defconfig
index b571d084c148..304673817ef1 100644
--- a/arch/powerpc/configs/pseries_defconfig
+++ b/arch/powerpc/configs/pseries_defconfig
@@ -315,7 +315,7 @@ CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_DEV_NX=y
CONFIG_CRYPTO_DEV_NX_ENCRYPT=m
-CONFIG_CRYPTO_DEV_VMX=y
+CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m
CONFIG_VIRTUALIZATION=y
CONFIG_KVM_BOOK3S_64=m
CONFIG_KVM_BOOK3S_64_HV=m
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 4f705674f94f..956f956607a5 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -761,12 +761,6 @@ config CRYPTO_DEV_QCOM_RNG
To compile this driver as a module, choose M here. The
module will be called qcom-rng. If unsure, say N.
-config CRYPTO_DEV_VMX
- bool "Support for VMX cryptographic acceleration instructions"
- depends on PPC64 && VSX
- help
- Support for VMX cryptographic acceleration instructions.
-
source "drivers/crypto/vmx/Kconfig"
config CRYPTO_DEV_IMGTEC_HASH
diff --git a/drivers/crypto/vmx/Kconfig b/drivers/crypto/vmx/Kconfig
index c85fab7ef0bd..1a3808b719f3 100644
--- a/drivers/crypto/vmx/Kconfig
+++ b/drivers/crypto/vmx/Kconfig
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
config CRYPTO_DEV_VMX_ENCRYPT
- tristate "Encryption acceleration support on P8 CPU"
- depends on CRYPTO_DEV_VMX
+ tristate "Power VMX cryptographic acceleration instructions driver"
+ depends on PPC64 && VSX
select CRYPTO_GHASH
default m
help
--
2.35.1
^ permalink raw reply related
* [PATCH v2 2/2] crypto: vmx: Add missing dependencies
From: Petr Vorel @ 2022-02-15 18:59 UTC (permalink / raw)
To: linux-crypto
Cc: Herbert Xu, Nicolai Stange, linux-kbuild, Nayna Jain, Petr Vorel,
Paulo Flabiano Smorigo, leitao, linuxppc-dev
In-Reply-To: <20220215185936.15576-1-pvorel@suse.cz>
vmx-crypto module depends on CRYPTO_AES, CRYPTO_CBC, CRYPTO_CTR or
CRYPTO_XTS, thus add them.
These dependencies are likely to be enabled, but if
CRYPTO_DEV_VMX_ENCRYPT=y && !CRYPTO_MANAGER_DISABLE_TESTS
and either of CRYPTO_AES, CRYPTO_CBC, CRYPTO_CTR or CRYPTO_XTS is built
as module or disabled, alg_test() from crypto/testmgr.c complains during
boot about failing to allocate the generic fallback implementations
(2 == ENOENT):
[ 0.540953] Failed to allocate xts(aes) fallback: -2
[ 0.541014] alg: skcipher: failed to allocate transform for p8_aes_xts: -2
[ 0.541120] alg: self-tests for p8_aes_xts (xts(aes)) failed (rc=-2)
[ 0.544440] Failed to allocate ctr(aes) fallback: -2
[ 0.544497] alg: skcipher: failed to allocate transform for p8_aes_ctr: -2
[ 0.544603] alg: self-tests for p8_aes_ctr (ctr(aes)) failed (rc=-2)
[ 0.547992] Failed to allocate cbc(aes) fallback: -2
[ 0.548052] alg: skcipher: failed to allocate transform for p8_aes_cbc: -2
[ 0.548156] alg: self-tests for p8_aes_cbc (cbc(aes)) failed (rc=-2)
[ 0.550745] Failed to allocate transformation for 'aes': -2
[ 0.550801] alg: cipher: Failed to load transform for p8_aes: -2
[ 0.550892] alg: self-tests for p8_aes (aes) failed (rc=-2)
Fixes: c07f5d3da643 ("crypto: vmx - Adding support for XTS")
Fixes: d2e3ae6f3aba ("crypto: vmx - Enabling VMX module for PPC64")
Suggested-by: Nicolai Stange <nstange@suse.de>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
changes v1->v2:
* use "select" instead of "depends on" (Nicolai)
* drop !CRYPTO_MANAGER_DISABLE_TESTS as the dependency is always (Nicolai)
drivers/crypto/vmx/Kconfig | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/crypto/vmx/Kconfig b/drivers/crypto/vmx/Kconfig
index 1a3808b719f3..ce054e64b92c 100644
--- a/drivers/crypto/vmx/Kconfig
+++ b/drivers/crypto/vmx/Kconfig
@@ -2,7 +2,11 @@
config CRYPTO_DEV_VMX_ENCRYPT
tristate "Power VMX cryptographic acceleration instructions driver"
depends on PPC64 && VSX
+ select CRYPTO_AES
+ select CRYPTO_CBC
+ select CRYPTO_CTR
select CRYPTO_GHASH
+ select CRYPTO_XTS
default m
help
Support for VMX cryptographic acceleration instructions on Power8 CPU.
--
2.35.1
^ permalink raw reply related
* [PATCH v2 0/2] vmx-crypto: Add missing dependencies
From: Petr Vorel @ 2022-02-15 18:59 UTC (permalink / raw)
To: linux-crypto
Cc: Herbert Xu, Nicolai Stange, linux-kbuild, Nayna Jain, Petr Vorel,
Paulo Flabiano Smorigo, leitao, linuxppc-dev
Hi,
[ Cc powerpc list and VMX people this time ]
changes v1->v2:
* new commit: crypto: vmx: Turn CRYPTO_DEV_VMX_ENCRYPT into tristate
* use "select" instead of "depends on" (Nicolai)
* drop !CRYPTO_MANAGER_DISABLE_TESTS as the dependency is always (Nicolai)
Petr Vorel (2):
crypto: vmx: Turn CRYPTO_DEV_VMX_ENCRYPT into tristate
crypto: vmx: Add missing dependencies
MAINTAINERS | 2 +-
arch/powerpc/configs/powernv_defconfig | 2 +-
arch/powerpc/configs/ppc64_defconfig | 2 +-
arch/powerpc/configs/pseries_defconfig | 2 +-
drivers/crypto/Kconfig | 6 ------
drivers/crypto/vmx/Kconfig | 8 ++++++--
6 files changed, 10 insertions(+), 12 deletions(-)
--
2.35.1
^ permalink raw reply
* Re: [PATCH v3 01/10] powerpc/pseries/vas: Use common names in VAS capability structure
From: Haren Myneni @ 2022-02-15 18:32 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev, mpe
In-Reply-To: <1644804791.k88oankrue.astroid@bobo.none>
On Mon, 2022-02-14 at 12:14 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of January 22, 2022 5:54 am:
> > target/used/avail_creds provides credits usage to user space via
> > sysfs and the same interface can be used on PowerNV in future.
> > Remove "lpar" from these names so that applicable on both PowerVM
> > and PowerNV.
>
> But not in this series? This is just to save you having to do more
> renaming later?
Thanks for your review.
Yes, Removing _lpar_ in struct elements to make it clear so that can
easily add in sysfs patch.
>
> Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
>
> > Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> > ---
> > arch/powerpc/platforms/pseries/vas.c | 10 +++++-----
> > arch/powerpc/platforms/pseries/vas.h | 6 +++---
> > 2 files changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index d243ddc58827..c0737379cc7b 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -310,8 +310,8 @@ static struct vas_window
> > *vas_allocate_window(int vas_id, u64 flags,
> >
> > cop_feat_caps = &caps->caps;
> >
> > - if (atomic_inc_return(&cop_feat_caps->used_lpar_creds) >
> > - atomic_read(&cop_feat_caps->target_lpar_creds))
> > {
> > + if (atomic_inc_return(&cop_feat_caps->used_creds) >
> > + atomic_read(&cop_feat_caps->target_creds)) {
> > pr_err("Credits are not available to allocate
> > window\n");
> > rc = -EINVAL;
> > goto out;
> > @@ -385,7 +385,7 @@ static struct vas_window
> > *vas_allocate_window(int vas_id, u64 flags,
> > free_irq_setup(txwin);
> > h_deallocate_vas_window(txwin->vas_win.winid);
> > out:
> > - atomic_dec(&cop_feat_caps->used_lpar_creds);
> > + atomic_dec(&cop_feat_caps->used_creds);
> > kfree(txwin);
> > return ERR_PTR(rc);
> > }
> > @@ -445,7 +445,7 @@ static int vas_deallocate_window(struct
> > vas_window *vwin)
> > }
> >
> > list_del(&win->win_list);
> > - atomic_dec(&caps->used_lpar_creds);
> > + atomic_dec(&caps->used_creds);
> > mutex_unlock(&vas_pseries_mutex);
> >
> > put_vas_user_win_ref(&vwin->task_ref);
> > @@ -521,7 +521,7 @@ static int __init get_vas_capabilities(u8 feat,
> > enum vas_cop_feat_type type,
> > }
> > caps->max_lpar_creds = be16_to_cpu(hv_caps->max_lpar_creds);
> > caps->max_win_creds = be16_to_cpu(hv_caps->max_win_creds);
> > - atomic_set(&caps->target_lpar_creds,
> > + atomic_set(&caps->target_creds,
> > be16_to_cpu(hv_caps->target_lpar_creds));
> > if (feat == VAS_GZIP_DEF_FEAT) {
> > caps->def_lpar_creds = be16_to_cpu(hv_caps-
> > >def_lpar_creds);
> > diff --git a/arch/powerpc/platforms/pseries/vas.h
> > b/arch/powerpc/platforms/pseries/vas.h
> > index 4ecb3fcabd10..fa7ce74f1e49 100644
> > --- a/arch/powerpc/platforms/pseries/vas.h
> > +++ b/arch/powerpc/platforms/pseries/vas.h
> > @@ -72,9 +72,9 @@ struct vas_cop_feat_caps {
> > };
> > /* Total LPAR available credits. Can be different from max LPAR
> > */
> > /* credits due to DLPAR operation */
> > - atomic_t target_lpar_creds;
> > - atomic_t used_lpar_creds; /* Used credits so far */
> > - u16 avail_lpar_creds; /* Remaining available credits */
> > + atomic_t target_creds;
> > + atomic_t used_creds; /* Used credits so far */
> > + u16 avail_creds; /* Remaining available credits */
> > };
> >
> > /*
> > --
> > 2.27.0
> >
> >
> >
^ permalink raw reply
* [PATCH v1 4/4] powerpc/ftrace: Style cleanup in ftrace_mprofile.S
From: Christophe Leroy @ 2022-02-15 18:31 UTC (permalink / raw)
To: Steven Rostedt, Ingo Molnar, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman
Cc: Naveen N . Rao, linuxppc-dev, linux-kernel
In-Reply-To: <ff535e86d3a69376a6d89168511d4e403835f18b.1644949750.git.christophe.leroy@csgroup.eu>
Add some line breaks to better match the file's style, add
some space after comma and fix a couple of misplaced blanks.
Suggested-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/trace/ftrace_mprofile.S | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kernel/trace/ftrace_mprofile.S b/arch/powerpc/kernel/trace/ftrace_mprofile.S
index eb077270ec2f..89639e64acd1 100644
--- a/arch/powerpc/kernel/trace/ftrace_mprofile.S
+++ b/arch/powerpc/kernel/trace/ftrace_mprofile.S
@@ -87,8 +87,9 @@ _GLOBAL(ftrace_regs_caller)
#endif
#ifdef CONFIG_LIVEPATCH_64
- mr r14,r7 /* remember old NIP */
+ mr r14, r7 /* remember old NIP */
#endif
+
/* Calculate ip from nip-4 into r3 for call below */
subi r3, r7, MCOUNT_INSN_SIZE
@@ -102,7 +103,7 @@ _GLOBAL(ftrace_regs_caller)
PPC_STL r11, _CCR(r1)
/* Load &pt_regs in r6 for call below */
- addi r6, r1 ,STACK_FRAME_OVERHEAD
+ addi r6, r1, STACK_FRAME_OVERHEAD
/* ftrace_call(r3, r4, r5, r6) */
.globl ftrace_regs_call
@@ -113,6 +114,7 @@ ftrace_regs_call:
/* Load ctr with the possibly modified NIP */
PPC_LL r3, _NIP(r1)
mtctr r3
+
#ifdef CONFIG_LIVEPATCH_64
cmpd r14, r3 /* has NIP been altered? */
#endif
@@ -196,7 +198,7 @@ _GLOBAL(ftrace_caller)
#ifdef CONFIG_LIVEPATCH_64
SAVE_GPR(14, r1)
- mr r14,r7 /* remember old NIP */
+ mr r14, r7 /* remember old NIP */
#endif
/* Calculate ip from nip-4 into r3 for call below */
subi r3, r7, MCOUNT_INSN_SIZE
@@ -210,7 +212,7 @@ _GLOBAL(ftrace_caller)
PPC_STL r8, _MSR(r1)
/* Load &pt_regs in r6 for call below */
- addi r6, r1 ,STACK_FRAME_OVERHEAD
+ addi r6, r1, STACK_FRAME_OVERHEAD
/* ftrace_call(r3, r4, r5, r6) */
.globl ftrace_call
@@ -220,6 +222,7 @@ ftrace_call:
PPC_LL r3, _NIP(r1)
mtctr r3
+
#ifdef CONFIG_LIVEPATCH_64
cmpd r14, r3 /* has NIP been altered? */
REST_GPR(14, r1)
@@ -244,6 +247,7 @@ ftrace_call:
/* Based on the cmpd above, if the NIP was altered handle livepatch */
bne- livepatch_handler
#endif
+
bctr /* jump after _mcount site */
#ifdef CONFIG_LIVEPATCH_64
--
2.34.1
^ permalink raw reply related
* [PATCH v1 3/4] powerpc/ftrace: Have arch_ftrace_get_regs() return NULL unless FL_SAVE_REGS is set
From: Christophe Leroy @ 2022-02-15 18:31 UTC (permalink / raw)
To: Steven Rostedt, Ingo Molnar, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman
Cc: Naveen N . Rao, linuxppc-dev, linux-kernel
In-Reply-To: <ff535e86d3a69376a6d89168511d4e403835f18b.1644949750.git.christophe.leroy@csgroup.eu>
When FL_SAVE_REGS is not set we get here via ftrace_caller()
which doesn't save all registers.
ftrace_caller() explicitely clears regs.msr, so we can rely
on it to know where we come from. We don't expect MSR register
to be 0 at all when involving ftrace.
Reported-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Fixes: 40b035efe288 ("powerpc/ftrace: Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/ftrace.h | 3 ++-
arch/powerpc/kernel/trace/ftrace_mprofile.S | 4 ++++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h
index 70b457097098..ff034ae4e472 100644
--- a/arch/powerpc/include/asm/ftrace.h
+++ b/arch/powerpc/include/asm/ftrace.h
@@ -30,7 +30,8 @@ struct ftrace_regs {
static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *fregs)
{
- return &fregs->regs;
+ /* We clear regs.msr in ftrace_call */
+ return fregs->regs.msr ? &fregs->regs : NULL;
}
static __always_inline void ftrace_instruction_pointer_set(struct ftrace_regs *fregs,
diff --git a/arch/powerpc/kernel/trace/ftrace_mprofile.S b/arch/powerpc/kernel/trace/ftrace_mprofile.S
index 8443902d5a05..eb077270ec2f 100644
--- a/arch/powerpc/kernel/trace/ftrace_mprofile.S
+++ b/arch/powerpc/kernel/trace/ftrace_mprofile.S
@@ -205,6 +205,10 @@ _GLOBAL(ftrace_caller)
PPC_STL r0, _LINK(r1)
mr r4, r0
+ /* Clear MSR to flag as ftrace_caller versus frace_regs_caller */
+ li r8, 0
+ PPC_STL r8, _MSR(r1)
+
/* Load &pt_regs in r6 for call below */
addi r6, r1 ,STACK_FRAME_OVERHEAD
--
2.34.1
^ permalink raw reply related
* [PATCH v1 2/4] powerpc/ftrace: Add recursion protection in prepare_ftrace_return()
From: Christophe Leroy @ 2022-02-15 18:31 UTC (permalink / raw)
To: Steven Rostedt, Ingo Molnar, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman
Cc: Naveen N . Rao, linuxppc-dev, linux-kernel
In-Reply-To: <ff535e86d3a69376a6d89168511d4e403835f18b.1644949750.git.christophe.leroy@csgroup.eu>
The function_graph_enter() does not provide any recursion protection.
Add a protection in prepare_ftrace_return() in case
function_graph_enter() calls something that gets
function graph traced.
Reported-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Fixes: 830213786c49 ("powerpc/ftrace: directly call of function graph tracer by ftrace caller")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/trace/ftrace.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c
index 74a176e394ef..f21b8fbd418e 100644
--- a/arch/powerpc/kernel/trace/ftrace.c
+++ b/arch/powerpc/kernel/trace/ftrace.c
@@ -944,6 +944,7 @@ unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip,
unsigned long sp)
{
unsigned long return_hooker;
+ int bit;
if (unlikely(ftrace_graph_is_dead()))
goto out;
@@ -951,10 +952,16 @@ unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip,
if (unlikely(atomic_read(¤t->tracing_graph_pause)))
goto out;
+ bit = ftrace_test_recursion_trylock(ip, parent);
+ if (bit < 0)
+ goto out;
+
return_hooker = ppc_function_entry(return_to_handler);
if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp))
parent = return_hooker;
+
+ ftrace_test_recursion_unlock(bit);
out:
return parent;
}
--
2.34.1
^ permalink raw reply related
* [PATCH v1 1/4] powerpc/ftrace: Also save r1 in ftrace_caller()
From: Christophe Leroy @ 2022-02-15 18:31 UTC (permalink / raw)
To: Steven Rostedt, Ingo Molnar, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman
Cc: Naveen N . Rao, linuxppc-dev, linux-kernel
Also save r1 in ftrace_caller()
r1 is needed during unwinding when the function_graph tracer
is active.
Reported-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Fixes: 830213786c49 ("powerpc/ftrace: directly call of function graph tracer by ftrace caller")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/trace/ftrace_mprofile.S | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/powerpc/kernel/trace/ftrace_mprofile.S b/arch/powerpc/kernel/trace/ftrace_mprofile.S
index 56da60e98327..8443902d5a05 100644
--- a/arch/powerpc/kernel/trace/ftrace_mprofile.S
+++ b/arch/powerpc/kernel/trace/ftrace_mprofile.S
@@ -173,6 +173,10 @@ _GLOBAL(ftrace_caller)
beq ftrace_no_trace
#endif
+ /* Save previous stack pointer (r1) */
+ addi r8, r1, SWITCH_FRAME_SIZE
+ PPC_STL r8, GPR1(r1)
+
/* Get the _mcount() call site out of LR */
mflr r7
PPC_STL r7, _NIP(r1)
--
2.34.1
^ permalink raw reply related
* Re: [PATCH v2 09/13] powerpc/ftrace: Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS
From: Naveen N. Rao @ 2022-02-15 16:26 UTC (permalink / raw)
To: Steven Rostedt
Cc: Petr Mladek, Joe Lawrence, Jiri Kosina,
linux-kernel@vger.kernel.org, Ingo Molnar, Masami Hiramatsu,
Josh Poimboeuf, live-patching@vger.kernel.org, Miroslav Benes,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <20220215093849.556d5444@gandalf.local.home>
Steven Rostedt wrote:
> On Tue, 15 Feb 2022 19:06:48 +0530
> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> wrote:
>
>> As I understand it, the reason ftrace_get_regs() was introduced was to
>> be able to only return the pt_regs, if _all_ registers were saved into
>> it, which we don't do when coming in through ftrace_caller(). See the
>> x86 implementation (commit 02a474ca266a47 ("ftrace/x86: Allow for
>> arguments to be passed in to ftrace_regs by default"), which returns
>> pt_regs conditionally.
>
> I can give you the history of ftrace_caller and ftrace_regs_caller.
>
> ftrace_caller saved just enough as was denoted for gcc mcount trampolines.
> The new fentry which happens at the start of the function, whereas mcount
> happens after the stack frame is set up, may change the rules on some
> architectures.
>
> As for ftrace_regs_caller, that was created for kprobes. As the majority of
> kprobes were added at the start of the function, it made sense to hook into
> ftrace as the ftrace trampoline call is much faster than taking a
> breakpoint interrupt. But to keep compatibility with breakpoint
> interrupts, we needed to fill in all the registers, and make it act just
> like a breakpoint interrupt.
>
> I've been wanting to record function parameters, and because the ftrace
> trampoline must at a minimum save the function parameters before calling
> the ftrace callbacks, all the information for those parameters were being
> saved but were never exposed to the ftrace callbacks. I created the the
> DYNAMIC_FTRACE_WITH_ARGS to expose them. I first just used pt_regs with
> just the parameters filled in, but that was criticized as it could be
> confusing where the non filled in pt_regs might be used and thinking they
> are legitimate. So I created ftrace_regs that would give you just the
> function arguments (if DYNAMIC_FTRACE_WITH_ARGS is defined), or it will
> give you a full pt_regs, if the caller came from the ftrace_regs_caller. If
> not, it will give you a NULL pointer.
>
> The first user to use the args was live kernel patching, as they only need
> that and the return pointer.
Thanks, that helps.
- Naveen
^ permalink raw reply
* Re: [PATCH v2 09/13] powerpc/ftrace: Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS
From: Naveen N. Rao @ 2022-02-15 16:25 UTC (permalink / raw)
To: Christophe Leroy, Vasily Gorbik, Heiko Carstens, Jiri Kosina,
Joe Lawrence, Josh Poimboeuf, Miroslav Benes, Ingo Molnar,
Michael Ellerman, Petr Mladek, Steven Rostedt
Cc: linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux-kernel@vger.kernel.org, live-patching@vger.kernel.org
In-Reply-To: <5c7b5334-6071-f131-a509-9a49ca3d628c@csgroup.eu>
Christophe Leroy wrote:
> + S390 people
>
> Le 15/02/2022 à 15:28, Christophe Leroy a écrit :
>>
>>
>> Le 15/02/2022 à 14:36, Naveen N. Rao a écrit :
>>> Michael Ellerman wrote:
>>>> Christophe Leroy <christophe.leroy@csgroup.eu> writes:
>>>>> Le 14/02/2022 à 16:25, Naveen N. Rao a écrit :
>>>>>> Christophe Leroy wrote:
>>>>>>> Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS. It accelerates the call
>>>>>>> of livepatching.
>>>>>>>
>>>>>>> Also note that powerpc being the last one to convert to
>>>>>>> CONFIG_DYNAMIC_FTRACE_WITH_ARGS, it will now be possible to remove
>>>>>>> klp_arch_set_pc() on all architectures.
>>>>>>>
>>>>>>> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
>>>>>>> ---
>>>>>>> arch/powerpc/Kconfig | 1 +
>>>>>>> arch/powerpc/include/asm/ftrace.h | 17 +++++++++++++++++
>>>>>>> arch/powerpc/include/asm/livepatch.h | 4 +---
>>>>>>> 3 files changed, 19 insertions(+), 3 deletions(-)
>>>>>>>
>>>>>>> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
>>>>>>> index cdac2115eb00..e2b1792b2aae 100644
>>>>>>> --- a/arch/powerpc/Kconfig
>>>>>>> +++ b/arch/powerpc/Kconfig
>>>>>>> @@ -210,6 +210,7 @@ config PPC
>>>>>>> select HAVE_DEBUG_KMEMLEAK
>>>>>>> select HAVE_DEBUG_STACKOVERFLOW
>>>>>>> select HAVE_DYNAMIC_FTRACE
>>>>>>> + select HAVE_DYNAMIC_FTRACE_WITH_ARGS if MPROFILE_KERNEL ||
>>>>>>> PPC32
>>>>>>> select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL ||
>>>>>>> PPC32
>>>>>>> select HAVE_EBPF_JIT
>>>>>>> select HAVE_EFFICIENT_UNALIGNED_ACCESS if
>>>>>>> !(CPU_LITTLE_ENDIAN && POWER7_CPU)
>>>>>>> diff --git a/arch/powerpc/include/asm/ftrace.h
>>>>>>> b/arch/powerpc/include/asm/ftrace.h
>>>>>>> index b3f6184f77ea..45c3d6f11daa 100644
>>>>>>> --- a/arch/powerpc/include/asm/ftrace.h
>>>>>>> +++ b/arch/powerpc/include/asm/ftrace.h
>>>>>>> @@ -22,6 +22,23 @@ static inline unsigned long
>>>>>>> ftrace_call_adjust(unsigned long addr)
>>>>>>> struct dyn_arch_ftrace {
>>>>>>> struct module *mod;
>>>>>>> };
>>>>>>> +
>>>>>>> +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
>>>>>>> +struct ftrace_regs {
>>>>>>> + struct pt_regs regs;
>>>>>>> +};
>>>>>>> +
>>>>>>> +static __always_inline struct pt_regs
>>>>>>> *arch_ftrace_get_regs(struct ftrace_regs *fregs)
>>>>>>> +{
>>>>>>> + return &fregs->regs;
>>>>>>> +}
>>>>>>
>>>>>> I think this is wrong. We need to differentiate between
>>>>>> ftrace_caller() and ftrace_regs_caller() here, and only return
>>>>>> pt_regs if coming in through ftrace_regs_caller() (i.e.,
>>>>>> FL_SAVE_REGS is set).
>>>>>
>>>>> Not sure I follow you.
>>>>>
>>>>> This is based on 5740a7c71ab6 ("s390/ftrace: add
>>>>> HAVE_DYNAMIC_FTRACE_WITH_ARGS support")
>>>>>
>>>>> It's all the point of HAVE_DYNAMIC_FTRACE_WITH_ARGS, have the regs
>>>>> also with ftrace_caller().
>>>>>
>>>>> Sure you only have the params, but that's the same on s390, so what
>>>>> did I miss ?
Steven has explained the rationale for this in his other response:
https://lore.kernel.org/all/20220215093849.556d5444@gandalf.local.home/
>>>
>>> It looks like s390 is special since it apparently saves all registers
>>> even for ftrace_caller:
>>> https://lore.kernel.org/all/YbipdU5X4HNDWIni@osiris/
>>
>> It is not what I understand from their code, see
>> https://elixir.bootlin.com/linux/v5.17-rc3/source/arch/s390/kernel/mcount.S#L37
>>
>>
>> They have a common macro called with argument 'allregs' which is set to
>> 0 for ftrace_caller() and 1 for ftrace_regs_caller().
>> When allregs == 1, the macro seems to save more.
>>
>> But ok, I can do like x86, but I need a trick to know whether
>> FL_SAVE_REGS is set or not, like they do with fregs->regs.cs
>> Any idea what the condition can be for powerpc ?
We'll need to explicitly zero-out something in pt_regs in
ftrace_caller(). We can probably use regs->msr since we don't expect it
to be zero when saved from ftrace_regs_caller().
>>
>
> Finally, it looks like this change is done via commit 894979689d3a
> ("s390/ftrace: provide separate ftrace_caller/ftrace_regs_caller
> implementations") four hours the same day after the implementation of
> arch_ftrace_get_regs()
>
> They may have forgotten to change arch_ftrace_get_regs() which was added
> in commit 5740a7c71ab6 ("s390/ftrace: add HAVE_DYNAMIC_FTRACE_WITH_ARGS
> support") with the assumption that ftrace_caller and ftrace_regs_caller
> where identical.
Indeed, good find!
Thanks,
Naveen
^ permalink raw reply
* Re: [PATCH v4 00/13] Fix LKDTM for PPC64/IA64/PARISC v4
From: Kees Cook @ 2022-02-15 16:07 UTC (permalink / raw)
To: Christophe Leroy
Cc: linux-arch, linux-ia64, linux-parisc, Arnd Bergmann,
Greg Kroah-Hartman, Helge Deller, linux-kernel,
James E.J. Bottomley, linux-mm, Paul Mackerras, Andrew Morton,
linuxppc-dev
In-Reply-To: <cover.1644928018.git.christophe.leroy@csgroup.eu>
On Tue, Feb 15, 2022 at 01:40:55PM +0100, Christophe Leroy wrote:
> PPC64/IA64/PARISC have function descriptors. LKDTM doesn't work
> on those three architectures because LKDTM messes up function
> descriptors with functions.
>
> This series does some cleanup in the three architectures and
> refactors function descriptors so that it can then easily use it
> in a generic way in LKDTM.
Thanks for doing this! It looks good to me. :)
-Kees
>
> Changes in v4:
> - Added patch 1 which Fixes 'sparse' for powerpc64le after wrong report on previous series, refer https://github.com/ruscur/linux-ci/actions/runs/1351427671
> - Exported dereference_function_descriptor() to modules
> - Addressed other received comments
> - Rebased on latest powerpc/next (5a72345e6a78120368fcc841b570331b6c5a50da)
>
> Changes in v3:
> - Addressed received comments
> - Swapped some of the powerpc patches to keep func_descr_t renamed as struct func_desc and remove 'struct ppc64_opd_entry'
> - Changed HAVE_FUNCTION_DESCRIPTORS macro to a config item CONFIG_HAVE_FUNCTION_DESCRIPTORS
> - Dropped patch 11 ("Fix lkdtm_EXEC_RODATA()")
>
> Changes in v2:
> - Addressed received comments
> - Moved dereference_[kernel]_function_descriptor() out of line
> - Added patches to remove func_descr_t and func_desc_t in powerpc
> - Using func_desc_t instead of funct_descr_t
> - Renamed HAVE_DEREFERENCE_FUNCTION_DESCRIPTOR to HAVE_FUNCTION_DESCRIPTORS
> - Added a new lkdtm test to check protection of function descriptors
>
> Christophe Leroy (13):
> powerpc: Fix 'sparse' checking on PPC64le
> powerpc: Move and rename func_descr_t
> powerpc: Use 'struct func_desc' instead of 'struct ppc64_opd_entry'
> powerpc: Remove 'struct ppc64_opd_entry'
> powerpc: Prepare func_desc_t for refactorisation
> ia64: Rename 'ip' to 'addr' in 'struct fdesc'
> asm-generic: Define CONFIG_HAVE_FUNCTION_DESCRIPTORS
> asm-generic: Define 'func_desc_t' to commonly describe function
> descriptors
> asm-generic: Refactor dereference_[kernel]_function_descriptor()
> lkdtm: Force do_nothing() out of line
> lkdtm: Really write into kernel text in WRITE_KERN
> lkdtm: Fix execute_[user]_location()
> lkdtm: Add a test for function descriptors protection
>
> arch/Kconfig | 3 +
> arch/ia64/Kconfig | 1 +
> arch/ia64/include/asm/elf.h | 2 +-
> arch/ia64/include/asm/sections.h | 24 +-------
> arch/ia64/kernel/module.c | 6 +-
> arch/parisc/Kconfig | 1 +
> arch/parisc/include/asm/sections.h | 16 ++----
> arch/parisc/kernel/process.c | 21 -------
> arch/powerpc/Kconfig | 1 +
> arch/powerpc/Makefile | 2 +-
> arch/powerpc/include/asm/code-patching.h | 2 +-
> arch/powerpc/include/asm/elf.h | 6 ++
> arch/powerpc/include/asm/sections.h | 29 ++--------
> arch/powerpc/include/asm/types.h | 6 --
> arch/powerpc/include/uapi/asm/elf.h | 8 ---
> arch/powerpc/kernel/module_64.c | 42 ++++++--------
> arch/powerpc/kernel/ptrace/ptrace.c | 6 ++
> arch/powerpc/kernel/signal_64.c | 8 +--
> drivers/misc/lkdtm/core.c | 1 +
> drivers/misc/lkdtm/lkdtm.h | 1 +
> drivers/misc/lkdtm/perms.c | 71 +++++++++++++++++++-----
> include/asm-generic/sections.h | 15 ++++-
> include/linux/kallsyms.h | 2 +-
> kernel/extable.c | 24 +++++++-
> tools/testing/selftests/lkdtm/tests.txt | 1 +
> 25 files changed, 155 insertions(+), 144 deletions(-)
>
> --
> 2.34.1
>
--
Kees Cook
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox