[PATCH vfio 0/3] pds/vfio: Fixes for locking bugs

[PATCH vfio 0/3] pds/vfio: Fixes for locking bugs

[Brett Creeley]

This series contains fixes for locking bugs in the recently introduced
pds-vfio-pci driver. There was an initial bug reported by Dan Carpenter
at:

https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/

However, more locking bugs were found when looking into the previously
mentioned issue. So, all fixes are included in this series.

Brett Creeley (3):
  pds/vfio: Fix spinlock bad magic BUG
  pds/vfio: Fix mutex lock->magic != lock warning
  pds/vfio: Fix possible sleep while in atomic context

 drivers/vfio/pci/pds/vfio_dev.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

-- 
2.17.1

[PATCH vfio 1/3] pds/vfio: Fix spinlock bad magic BUG

[Brett Creeley]

The following BUG was found when running on a kernel with
CONFIG_DEBUG_SPINLOCK=y set:

[  675.116953] BUG: spinlock bad magic on CPU#2, bash/2481
[  675.116966]  lock: 0xffff8d6052a88f50, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
[  675.116978] CPU: 2 PID: 2481 Comm: bash Tainted: G S                 6.6.0-rc1-next-20230911 #1
[  675.116986] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
[  675.116991] Call Trace:
[  675.116997]  <TASK>
[  675.117002]  dump_stack_lvl+0x36/0x50
[  675.117014]  do_raw_spin_lock+0x79/0xc0
[  675.117032]  pds_vfio_reset+0x1d/0x60 [pds_vfio_pci]
[  675.117049]  pci_reset_function+0x4b/0x70
[  675.117061]  reset_store+0x5b/0xa0
[  675.117074]  kernfs_fop_write_iter+0x137/0x1d0
[  675.117087]  vfs_write+0x2de/0x410
[  675.117101]  ksys_write+0x5d/0xd0
[  675.117111]  do_syscall_64+0x3b/0x90
[  675.117122]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  675.117135] RIP: 0033:0x7f9ebbd1fa28
[  675.117141] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[  675.117148] RSP: 002b:00007ffdff410728 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  675.117156] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f9ebbd1fa28
[  675.117161] RDX: 0000000000000002 RSI: 000055ffc5fdf7c0 RDI: 0000000000000001
[  675.117166] RBP: 000055ffc5fdf7c0 R08: 000000000000000a R09: 00007f9ebbd7fae0
[  675.117170] R10: 000000000000000a R11: 0000000000000246 R12: 00007f9ebbfc06e0
[  675.117174] R13: 0000000000000002 R14: 00007f9ebbfbb860 R15: 0000000000000002
[  675.117180]  </TASK>

As shown, the .magic: 00000000, does not match the expected value. This
is because spin_lock_init() is never called for the reset_lock. Fix
this by calling spin_lock_init(&pds_vfio->reset_lock) when initializing
the device.

Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index b46174f5eb09..147a543a7c39 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -155,6 +155,8 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 
 	pds_vfio->vf_id = vf_id;
 
+	spin_lock_init(&pds_vfio->reset_lock);
+
 	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
 	vdev->mig_ops = &pds_vfio_lm_ops;
 	vdev->log_ops = &pds_vfio_log_ops;
-- 
2.17.1

[PATCH vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning

[Brett Creeley]

The following BUG was found when running on a kernel with
CONFIG_DEBUG_MUTEXES=y set:

[  502.794510] DEBUG_LOCKS_WARN_ON(lock->magic != lock)
[  502.794519] WARNING: CPU: 16 PID: 2571 at kernel/locking/mutex.c:1085 mutex_trylock+0x10d/0x120
[  502.794537] Modules linked in: pds_vfio_pci(OE) pds_core xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_compat
nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl mei_me ses intel_cstate ipmi_ssif sunrpc enclosure mei nvme
intel_uncore nvme_core hpwdt hpilo ioatdma acpi_tad acpi_ipmi pcspkr lpc_ich intel_pch_thermal acpi_power_meter ipmi_si xfs
libcrc32c sd_mod t10_pi crc64_rocksoft crc64 sg mgag200 drm_kms_helper drm_shmem_helper crc32c_intel serio_raw igb smartpqi
dca drm scsi_transport_sas i2c_algo_bit ionic wmi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler
fuse [last unloaded: pds_core]
[  502.794673] CPU: 16 PID: 2571 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
[  502.794676] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
[  502.794678] RIP: 0010:mutex_trylock+0x10d/0x120
[  502.794681] Code: ff 85 c0 0f 84 40 ff ff ff 8b 3d f2 35 eb 00 85 ff 0f 85 32 ff ff ff 48 c7 c6 47 ff d3 bc 48 c7 c7 80 69 d3 bc e8 33 c7 4a ff <0f> 0b e9 18 ff ff ff b8 01 00 00 00 e9 5c ff ff ff 66 90 90 90 90
[  502.794684] RSP: 0018:ffffabadc750fda8 EFLAGS: 00010282
[  502.794686] RAX: 0000000000000000 RBX: ffff930000451000 RCX: 0000000000000027
[  502.794688] RDX: 0000000000000027 RSI: ffffabadc750fca0 RDI: ffff930f7f920948
[  502.794690] RBP: ffff930000451708 R08: 0000000000000000 R09: c0000000ffff7fff
[  502.794692] R10: 0000000000000001 R11: ffffabadc750fc40 R12: 0000000000000000
[  502.794694] R13: fffffffffffffff2 R14: ffffabadc750fea0 R15: ffff92ffc85dc520
[  502.794695] FS:  00007f3d8e322740(0000) GS:ffff930f7f900000(0000) knlGS:0000000000000000
[  502.794697] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  502.794698] CR2: 000055cc4f6d9310 CR3: 0000000109894005 CR4: 00000000007706e0
[  502.794700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  502.794701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  502.794703] PKRU: 55555554
[  502.794704] Call Trace:
[  502.794707]  <TASK>
[  502.794710]  ? __warn+0x85/0x140
[  502.794718]  ? mutex_trylock+0x10d/0x120
[  502.794720]  ? report_bug+0xfc/0x1e0
[  502.794727]  ? handle_bug+0x3f/0x70
[  502.794732]  ? exc_invalid_op+0x17/0x70
[  502.794735]  ? asm_exc_invalid_op+0x1a/0x20
[  502.794742]  ? mutex_trylock+0x10d/0x120
[  502.794744]  ? mutex_trylock+0x10d/0x120
[  502.794748]  pds_vfio_reset+0x3a/0x60 [pds_vfio_pci]
[  502.794756]  pci_reset_function+0x4b/0x70
[  502.794763]  reset_store+0x5b/0xa0
[  502.794770]  kernfs_fop_write_iter+0x137/0x1d0
[  502.794776]  vfs_write+0x2de/0x410
[  502.794784]  ksys_write+0x5d/0xd0
[  502.794787]  do_syscall_64+0x3b/0x90
[  502.794790]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  502.794794] RIP: 0033:0x7f3d8d51fa28
[  502.794796] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[  502.794799] RSP: 002b:00007ffc7b6c76f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  502.794801] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d8d51fa28
[  502.794803] RDX: 0000000000000002 RSI: 000055cc4f742aa0 RDI: 0000000000000001
[  502.794804] RBP: 000055cc4f742aa0 R08: 000000000000000a R09: 00007f3d8d57fae0
[  502.794805] R10: 000000000000000a R11: 0000000000000246 R12: 00007f3d8d7c06e0
[  502.794807] R13: 0000000000000002 R14: 00007f3d8d7bb860 R15: 0000000000000002
[  502.794809]  </TASK>

As shown, lock->magic != lock. This is because
mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So,
if a reset is initiated before the VFIO device is opened the mutex will
have never been initialized. Fix this by calling
mutex_init(&pds_vfio->state_mutex) in the VFIO init path.

Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index 147a543a7c39..9db5f2c8f1ea 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device *vdev)
 
 	pds_vfio->vf_id = vf_id;
 
+	mutex_init(&pds_vfio->state_mutex);
 	spin_lock_init(&pds_vfio->reset_lock);
 
 	vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
@@ -181,7 +182,6 @@ static int pds_vfio_open_device(struct vfio_device *vdev)
 	if (err)
 		return err;
 
-	mutex_init(&pds_vfio->state_mutex);
 	pds_vfio->state = VFIO_DEVICE_STATE_RUNNING;
 	pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING;
 
-- 
2.17.1

[PATCH vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Brett Creeley]

The driver could possibly sleep while in atomic context resulting
in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
set:

[  227.229806] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
[  227.229818] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
[  227.229824] preempt_count: 1, expected: 0
[  227.229827] RCU nest depth: 0, expected: 0
[  227.229832] CPU: 5 PID: 2817 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
[  227.229839] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
[  227.229843] Call Trace:
[  227.229848]  <TASK>
[  227.229853]  dump_stack_lvl+0x36/0x50
[  227.229865]  __might_resched+0x123/0x170
[  227.229877]  mutex_lock+0x1e/0x50
[  227.229891]  pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
[  227.229909]  pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
[  227.229923]  pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
[  227.229937]  pci_reset_function+0x4b/0x70
[  227.229948]  reset_store+0x5b/0xa0
[  227.229959]  kernfs_fop_write_iter+0x137/0x1d0
[  227.229972]  vfs_write+0x2de/0x410
[  227.229986]  ksys_write+0x5d/0xd0
[  227.229996]  do_syscall_64+0x3b/0x90
[  227.230004]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  227.230017] RIP: 0033:0x7fb202b1fa28
[  227.230023] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[  227.230028] RSP: 002b:00007fff6915fbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  227.230036] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb202b1fa28
[  227.230040] RDX: 0000000000000002 RSI: 000055f3834d5aa0 RDI: 0000000000000001
[  227.230044] RBP: 000055f3834d5aa0 R08: 000000000000000a R09: 00007fb202b7fae0
[  227.230047] R10: 000000000000000a R11: 0000000000000246 R12: 00007fb202dc06e0
[  227.230050] R13: 0000000000000002 R14: 00007fb202dbb860 R15: 0000000000000002
[  227.230056]  </TASK>

This can happen if pds_vfio_put_restore_file() and/or
pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
while the spin_lock(&pds_vfio->reset_lock) is held, which can
happen during while calling pds_vfio_state_mutex_unlock().

Fix this by releasing the spin_unlock(&pds_vfio->reset_lock) before
calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() and
re-acquiring spin_lock(&pds_vfio->reset_lock) after the previously
mentioned functions are called to protect setting the subsequent
state/deferred reset settings.

The only possible concerns are other threads that may call
pds_vfio_put_restore_file() and/or pds_vfio_put_save_file(). However,
those paths are already protected by the state mutex_lock().

Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
Closes: https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
---
 drivers/vfio/pci/pds/vfio_dev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
index 9db5f2c8f1ea..6e664cb05dd1 100644
--- a/drivers/vfio/pci/pds/vfio_dev.c
+++ b/drivers/vfio/pci/pds/vfio_dev.c
@@ -33,8 +33,10 @@ void pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio)
 	if (pds_vfio->deferred_reset) {
 		pds_vfio->deferred_reset = false;
 		if (pds_vfio->state == VFIO_DEVICE_STATE_ERROR) {
+			spin_unlock(&pds_vfio->reset_lock);
 			pds_vfio_put_restore_file(pds_vfio);
 			pds_vfio_put_save_file(pds_vfio);
+			spin_lock(&pds_vfio->reset_lock);
 			pds_vfio_dirty_disable(pds_vfio, false);
 		}
 		pds_vfio->state = pds_vfio->deferred_reset_state;
-- 
2.17.1

Re: [PATCH vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Alex Williamson]

On Thu, 14 Sep 2023 12:15:40 -0700
Brett Creeley <brett.creeley@amd.com> wrote:

> The driver could possibly sleep while in atomic context resulting
> in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
> set:
> 
> [  227.229806] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
> [  227.229818] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
> [  227.229824] preempt_count: 1, expected: 0
> [  227.229827] RCU nest depth: 0, expected: 0
> [  227.229832] CPU: 5 PID: 2817 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
> [  227.229839] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
> [  227.229843] Call Trace:
> [  227.229848]  <TASK>
> [  227.229853]  dump_stack_lvl+0x36/0x50
> [  227.229865]  __might_resched+0x123/0x170
> [  227.229877]  mutex_lock+0x1e/0x50
> [  227.229891]  pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
> [  227.229909]  pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
> [  227.229923]  pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
> [  227.229937]  pci_reset_function+0x4b/0x70
> [  227.229948]  reset_store+0x5b/0xa0
> [  227.229959]  kernfs_fop_write_iter+0x137/0x1d0
> [  227.229972]  vfs_write+0x2de/0x410
> [  227.229986]  ksys_write+0x5d/0xd0
> [  227.229996]  do_syscall_64+0x3b/0x90
> [  227.230004]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [  227.230017] RIP: 0033:0x7fb202b1fa28
> [  227.230023] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
> [  227.230028] RSP: 002b:00007fff6915fbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> [  227.230036] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb202b1fa28
> [  227.230040] RDX: 0000000000000002 RSI: 000055f3834d5aa0 RDI: 0000000000000001
> [  227.230044] RBP: 000055f3834d5aa0 R08: 000000000000000a R09: 00007fb202b7fae0
> [  227.230047] R10: 000000000000000a R11: 0000000000000246 R12: 00007fb202dc06e0
> [  227.230050] R13: 0000000000000002 R14: 00007fb202dbb860 R15: 0000000000000002
> [  227.230056]  </TASK>
> 
> This can happen if pds_vfio_put_restore_file() and/or
> pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
> while the spin_lock(&pds_vfio->reset_lock) is held, which can
> happen during while calling pds_vfio_state_mutex_unlock().
> 
> Fix this by releasing the spin_unlock(&pds_vfio->reset_lock) before
> calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() and
> re-acquiring spin_lock(&pds_vfio->reset_lock) after the previously
> mentioned functions are called to protect setting the subsequent
> state/deferred reset settings.
> 
> The only possible concerns are other threads that may call
> pds_vfio_put_restore_file() and/or pds_vfio_put_save_file(). However,
> those paths are already protected by the state mutex_lock().

Is there another viable solution to change reset_lock to a mutex?

I think this is the origin of this algorithm:

https://lore.kernel.org/all/20211019191025.GA4072278@nvidia.com/

But it's not clear to me why Jason chose an example with a spinlock and
if some subtlety here requires it.  Thanks,

Alex

> Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
> Closes: https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
> Signed-off-by: Brett Creeley <brett.creeley@amd.com>
> Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
>  drivers/vfio/pci/pds/vfio_dev.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
> index 9db5f2c8f1ea..6e664cb05dd1 100644
> --- a/drivers/vfio/pci/pds/vfio_dev.c
> +++ b/drivers/vfio/pci/pds/vfio_dev.c
> @@ -33,8 +33,10 @@ void pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio)
>  	if (pds_vfio->deferred_reset) {
>  		pds_vfio->deferred_reset = false;
>  		if (pds_vfio->state == VFIO_DEVICE_STATE_ERROR) {
> +			spin_unlock(&pds_vfio->reset_lock);
>  			pds_vfio_put_restore_file(pds_vfio);
>  			pds_vfio_put_save_file(pds_vfio);
> +			spin_lock(&pds_vfio->reset_lock);
>  			pds_vfio_dirty_disable(pds_vfio, false);
>  		}
>  		pds_vfio->state = pds_vfio->deferred_reset_state;

Re: [PATCH vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Brett Creeley]

On 9/14/2023 3:38 PM, Alex Williamson wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Thu, 14 Sep 2023 12:15:40 -0700
> Brett Creeley <brett.creeley@amd.com> wrote:
> 
>> The driver could possibly sleep while in atomic context resulting
>> in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
>> set:
>>
>> [  227.229806] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
>> [  227.229818] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
>> [  227.229824] preempt_count: 1, expected: 0
>> [  227.229827] RCU nest depth: 0, expected: 0
>> [  227.229832] CPU: 5 PID: 2817 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
>> [  227.229839] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
>> [  227.229843] Call Trace:
>> [  227.229848]  <TASK>
>> [  227.229853]  dump_stack_lvl+0x36/0x50
>> [  227.229865]  __might_resched+0x123/0x170
>> [  227.229877]  mutex_lock+0x1e/0x50
>> [  227.229891]  pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
>> [  227.229909]  pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
>> [  227.229923]  pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
>> [  227.229937]  pci_reset_function+0x4b/0x70
>> [  227.229948]  reset_store+0x5b/0xa0
>> [  227.229959]  kernfs_fop_write_iter+0x137/0x1d0
>> [  227.229972]  vfs_write+0x2de/0x410
>> [  227.229986]  ksys_write+0x5d/0xd0
>> [  227.229996]  do_syscall_64+0x3b/0x90
>> [  227.230004]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
>> [  227.230017] RIP: 0033:0x7fb202b1fa28
>> [  227.230023] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
>> [  227.230028] RSP: 002b:00007fff6915fbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>> [  227.230036] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb202b1fa28
>> [  227.230040] RDX: 0000000000000002 RSI: 000055f3834d5aa0 RDI: 0000000000000001
>> [  227.230044] RBP: 000055f3834d5aa0 R08: 000000000000000a R09: 00007fb202b7fae0
>> [  227.230047] R10: 000000000000000a R11: 0000000000000246 R12: 00007fb202dc06e0
>> [  227.230050] R13: 0000000000000002 R14: 00007fb202dbb860 R15: 0000000000000002
>> [  227.230056]  </TASK>
>>
>> This can happen if pds_vfio_put_restore_file() and/or
>> pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
>> while the spin_lock(&pds_vfio->reset_lock) is held, which can
>> happen during while calling pds_vfio_state_mutex_unlock().
>>
>> Fix this by releasing the spin_unlock(&pds_vfio->reset_lock) before
>> calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() and
>> re-acquiring spin_lock(&pds_vfio->reset_lock) after the previously
>> mentioned functions are called to protect setting the subsequent
>> state/deferred reset settings.
>>
>> The only possible concerns are other threads that may call
>> pds_vfio_put_restore_file() and/or pds_vfio_put_save_file(). However,
>> those paths are already protected by the state mutex_lock().
> 
> Is there another viable solution to change reset_lock to a mutex?
> 
> I think this is the origin of this algorithm:
> 
> https://lore.kernel.org/all/20211019191025.GA4072278@nvidia.com/
> 
> But it's not clear to me why Jason chose an example with a spinlock and
> if some subtlety here requires it.  Thanks,
> 
> Alex

It would be good to get some feedback from Jason on this before thinking 
about a different solution.

Thanks,

Brett

> 
>> Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
>> Closes: https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/
>> Signed-off-by: Brett Creeley <brett.creeley@amd.com>
>> Reviewed-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>>   drivers/vfio/pci/pds/vfio_dev.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c
>> index 9db5f2c8f1ea..6e664cb05dd1 100644
>> --- a/drivers/vfio/pci/pds/vfio_dev.c
>> +++ b/drivers/vfio/pci/pds/vfio_dev.c
>> @@ -33,8 +33,10 @@ void pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio)
>>        if (pds_vfio->deferred_reset) {
>>                pds_vfio->deferred_reset = false;
>>                if (pds_vfio->state == VFIO_DEVICE_STATE_ERROR) {
>> +                     spin_unlock(&pds_vfio->reset_lock);
>>                        pds_vfio_put_restore_file(pds_vfio);
>>                        pds_vfio_put_save_file(pds_vfio);
>> +                     spin_lock(&pds_vfio->reset_lock);
>>                        pds_vfio_dirty_disable(pds_vfio, false);
>>                }
>>                pds_vfio->state = pds_vfio->deferred_reset_state;
> 

Re: [PATCH vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Jason Gunthorpe]

On Thu, Sep 14, 2023 at 04:38:37PM -0600, Alex Williamson wrote:
> On Thu, 14 Sep 2023 12:15:40 -0700
> Brett Creeley <brett.creeley@amd.com> wrote:
> 
> > The driver could possibly sleep while in atomic context resulting
> > in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
> > set:
> > 
> > [  227.229806] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
> > [  227.229818] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
> > [  227.229824] preempt_count: 1, expected: 0
> > [  227.229827] RCU nest depth: 0, expected: 0
> > [  227.229832] CPU: 5 PID: 2817 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
> > [  227.229839] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
> > [  227.229843] Call Trace:
> > [  227.229848]  <TASK>
> > [  227.229853]  dump_stack_lvl+0x36/0x50
> > [  227.229865]  __might_resched+0x123/0x170
> > [  227.229877]  mutex_lock+0x1e/0x50
> > [  227.229891]  pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
> > [  227.229909]  pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
> > [  227.229923]  pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
> > [  227.229937]  pci_reset_function+0x4b/0x70
> > [  227.229948]  reset_store+0x5b/0xa0
> > [  227.229959]  kernfs_fop_write_iter+0x137/0x1d0
> > [  227.229972]  vfs_write+0x2de/0x410
> > [  227.229986]  ksys_write+0x5d/0xd0
> > [  227.229996]  do_syscall_64+0x3b/0x90
> > [  227.230004]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > [  227.230017] RIP: 0033:0x7fb202b1fa28
> > [  227.230023] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
> > [  227.230028] RSP: 002b:00007fff6915fbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > [  227.230036] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb202b1fa28
> > [  227.230040] RDX: 0000000000000002 RSI: 000055f3834d5aa0 RDI: 0000000000000001
> > [  227.230044] RBP: 000055f3834d5aa0 R08: 000000000000000a R09: 00007fb202b7fae0
> > [  227.230047] R10: 000000000000000a R11: 0000000000000246 R12: 00007fb202dc06e0
> > [  227.230050] R13: 0000000000000002 R14: 00007fb202dbb860 R15: 0000000000000002
> > [  227.230056]  </TASK>

I usually encourage people to trim the oops, remove the time stamp at least.
> > 
> > This can happen if pds_vfio_put_restore_file() and/or
> > pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
> > while the spin_lock(&pds_vfio->reset_lock) is held, which can
> > happen during while calling pds_vfio_state_mutex_unlock().
> > 
> > Fix this by releasing the spin_unlock(&pds_vfio->reset_lock) before
> > calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() and
> > re-acquiring spin_lock(&pds_vfio->reset_lock) after the previously
> > mentioned functions are called to protect setting the subsequent
> > state/deferred reset settings.
> > 
> > The only possible concerns are other threads that may call
> > pds_vfio_put_restore_file() and/or pds_vfio_put_save_file(). However,
> > those paths are already protected by the state mutex_lock().
> 
> Is there another viable solution to change reset_lock to a mutex?
> 
> I think this is the origin of this algorithm:
> 
> https://lore.kernel.org/all/20211019191025.GA4072278@nvidia.com/
> 
> But it's not clear to me why Jason chose an example with a spinlock and
> if some subtlety here requires it.  Thanks,

I think there was no specific reason it must be a spinlock

Certainly I'm not feeling comfortable just unlocking and relocking
like that. It would need a big explanation why it is safe in a
comment.

Jason

Re: [PATCH vfio 3/3] pds/vfio: Fix possible sleep while in atomic context

[Brett Creeley]

On 9/19/2023 11:59 AM, Jason Gunthorpe wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Thu, Sep 14, 2023 at 04:38:37PM -0600, Alex Williamson wrote:
>> On Thu, 14 Sep 2023 12:15:40 -0700
>> Brett Creeley <brett.creeley@amd.com> wrote:
>>
>>> The driver could possibly sleep while in atomic context resulting
>>> in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is
>>> set:
>>>
>>> [  227.229806] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
>>> [  227.229818] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash
>>> [  227.229824] preempt_count: 1, expected: 0
>>> [  227.229827] RCU nest depth: 0, expected: 0
>>> [  227.229832] CPU: 5 PID: 2817 Comm: bash Tainted: G S         OE      6.6.0-rc1-next-20230911 #1
>>> [  227.229839] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 01/23/2021
>>> [  227.229843] Call Trace:
>>> [  227.229848]  <TASK>
>>> [  227.229853]  dump_stack_lvl+0x36/0x50
>>> [  227.229865]  __might_resched+0x123/0x170
>>> [  227.229877]  mutex_lock+0x1e/0x50
>>> [  227.229891]  pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci]
>>> [  227.229909]  pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci]
>>> [  227.229923]  pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci]
>>> [  227.229937]  pci_reset_function+0x4b/0x70
>>> [  227.229948]  reset_store+0x5b/0xa0
>>> [  227.229959]  kernfs_fop_write_iter+0x137/0x1d0
>>> [  227.229972]  vfs_write+0x2de/0x410
>>> [  227.229986]  ksys_write+0x5d/0xd0
>>> [  227.229996]  do_syscall_64+0x3b/0x90
>>> [  227.230004]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
>>> [  227.230017] RIP: 0033:0x7fb202b1fa28
>>> [  227.230023] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 15 4d 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
>>> [  227.230028] RSP: 002b:00007fff6915fbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>>> [  227.230036] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb202b1fa28
>>> [  227.230040] RDX: 0000000000000002 RSI: 000055f3834d5aa0 RDI: 0000000000000001
>>> [  227.230044] RBP: 000055f3834d5aa0 R08: 000000000000000a R09: 00007fb202b7fae0
>>> [  227.230047] R10: 000000000000000a R11: 0000000000000246 R12: 00007fb202dc06e0
>>> [  227.230050] R13: 0000000000000002 R14: 00007fb202dbb860 R15: 0000000000000002
>>> [  227.230056]  </TASK>
> 
> I usually encourage people to trim the oops, remove the time stamp at least.

Makes sense. I will remember that going forward. Thanks for the suggestion.

Brett

>>>
>>> This can happen if pds_vfio_put_restore_file() and/or
>>> pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock)
>>> while the spin_lock(&pds_vfio->reset_lock) is held, which can
>>> happen during while calling pds_vfio_state_mutex_unlock().
>>>
>>> Fix this by releasing the spin_unlock(&pds_vfio->reset_lock) before
>>> calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() and
>>> re-acquiring spin_lock(&pds_vfio->reset_lock) after the previously
>>> mentioned functions are called to protect setting the subsequent
>>> state/deferred reset settings.
>>>
>>> The only possible concerns are other threads that may call
>>> pds_vfio_put_restore_file() and/or pds_vfio_put_save_file(). However,
>>> those paths are already protected by the state mutex_lock().
>>
>> Is there another viable solution to change reset_lock to a mutex?
>>
>> I think this is the origin of this algorithm:
>>
>> https://lore.kernel.org/all/20211019191025.GA4072278@nvidia.com/
>>
>> But it's not clear to me why Jason chose an example with a spinlock and
>> if some subtlety here requires it.  Thanks,
> 
> I think there was no specific reason it must be a spinlock
> 
> Certainly I'm not feeling comfortable just unlocking and relocking
> like that. It would need a big explanation why it is safe in a
> comment.

This follows the example in mlx5vf_state_mutex_unlock(), which releases 
the spinlock before calling mlx5vf_disable_fds().

However, there is a small difference where 
pds_vfio->deferred_reset_state could change in the window where the 
reset_lock isn't held. It seems this can be fixed this by a local 
deferred_reset_state in pds_vfio_state_mutex_unlock() that I set before 
unlocking to clear the fds.

Thanks,

Brett
> 
> Jason