[PATCH] Bluetooth: virtio_bt: fix device removal

[PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

Device removal is clearly out of virtio spec: it attempts to remove
unused buffers from a VQ before invoking device reset. To fix, make
open/close NOPs and do all cleanup/setup in probe/remove.

The cost here is a single skb wasted on an unused bt device - which
seems modest.

NB: with this fix in place driver still suffers from a race condition if
an interrupt triggers while device is being reset. Work on a fix for
that issue is in progress.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---

Note: completely untested, in particular the device isn't supported in QEMU.
Please do not queue directly - please help review and test and ack,
and I will queue this together with reset fixes.
Thanks!


 drivers/bluetooth/virtio_bt.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/bluetooth/virtio_bt.c b/drivers/bluetooth/virtio_bt.c
index 24a9258962fa..aea33ba9522c 100644
--- a/drivers/bluetooth/virtio_bt.c
+++ b/drivers/bluetooth/virtio_bt.c
@@ -50,8 +50,11 @@ static int virtbt_add_inbuf(struct virtio_bluetooth *vbt)
 
 static int virtbt_open(struct hci_dev *hdev)
 {
-	struct virtio_bluetooth *vbt = hci_get_drvdata(hdev);
+	return 0;
+}
 
+static int virtbt_open_vdev(struct virtio_bluetooth *vbt)
+{
 	if (virtbt_add_inbuf(vbt) < 0)
 		return -EIO;
 
@@ -61,7 +64,11 @@ static int virtbt_open(struct hci_dev *hdev)
 
 static int virtbt_close(struct hci_dev *hdev)
 {
-	struct virtio_bluetooth *vbt = hci_get_drvdata(hdev);
+	return 0;
+}
+
+static int virtbt_close_vdev(struct virtio_bluetooth *vbt)
+{
 	int i;
 
 	cancel_work_sync(&vbt->rx);
@@ -351,8 +358,14 @@ static int virtbt_probe(struct virtio_device *vdev)
 		goto failed;
 	}
 
+	virtio_device_ready(vdev);
+	if (virtbt_open_vdev(vbt))
+		goto open_failed;
+
 	return 0;
 
+open_failed:
+	hci_free_dev(hdev);
 failed:
 	vdev->config->del_vqs(vdev);
 	return err;
@@ -365,6 +378,7 @@ static void virtbt_remove(struct virtio_device *vdev)
 
 	hci_unregister_dev(hdev);
 	vdev->config->reset(vdev);
+	virtbt_close_vdev(vbt);
 
 	hci_free_dev(hdev);
 	vbt->hdev = NULL;
-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> Hi Michael,
> 
> > Device removal is clearly out of virtio spec: it attempts to remove
> > unused buffers from a VQ before invoking device reset. To fix, make
> > open/close NOPs and do all cleanup/setup in probe/remove.
> 
> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> to be doing. These are transport enable/disable callbacks from the BT
> Core towards the driver. It maps to a device being enabled/disabled by
> something like bluetoothd for example. So if disabled, I expect that no
> resources/queues are in use.
> 
> Maybe I misunderstand the virtio spec in that regard, but I would like
> to keep this fundamental concept of a Bluetooth driver. It does work
> with all other transports like USB, SDIO, UART etc.
> 
> > The cost here is a single skb wasted on an unused bt device - which
> > seems modest.
> 
> There should be no buffer used if the device is powered off. We also don’t
> have any USB URBs in-flight if the transport is not active.
> 
> > NB: with this fix in place driver still suffers from a race condition if
> > an interrupt triggers while device is being reset. Work on a fix for
> > that issue is in progress.
> 
> In the virtbt_close() callback we should deactivate all interrupts.
> 
> Regards
> 
> Marcel

If you want to do that then device has to be reset on close,
and fully reinitialized on open.
Can you work on a patch like that?
Given I don't have the device such a rework is probably more
than I can undertake.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Nov 25, 2021 at 10:01:25PM +0100, Marcel Holtmann wrote:
> Hi Michael,
> 
> >>> Device removal is clearly out of virtio spec: it attempts to remove
> >>> unused buffers from a VQ before invoking device reset. To fix, make
> >>> open/close NOPs and do all cleanup/setup in probe/remove.
> >> 
> >> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >> to be doing. These are transport enable/disable callbacks from the BT
> >> Core towards the driver. It maps to a device being enabled/disabled by
> >> something like bluetoothd for example. So if disabled, I expect that no
> >> resources/queues are in use.
> >> 
> >> Maybe I misunderstand the virtio spec in that regard, but I would like
> >> to keep this fundamental concept of a Bluetooth driver. It does work
> >> with all other transports like USB, SDIO, UART etc.
> >> 
> >>> The cost here is a single skb wasted on an unused bt device - which
> >>> seems modest.
> >> 
> >> There should be no buffer used if the device is powered off. We also don’t
> >> have any USB URBs in-flight if the transport is not active.
> >> 
> >>> NB: with this fix in place driver still suffers from a race condition if
> >>> an interrupt triggers while device is being reset. Work on a fix for
> >>> that issue is in progress.
> >> 
> >> In the virtbt_close() callback we should deactivate all interrupts.
> >> 
> > 
> > If you want to do that then device has to be reset on close,
> > and fully reinitialized on open.
> > Can you work on a patch like that?
> > Given I don't have the device such a rework is probably more
> > than I can undertake.
> 
> so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
> virtbt_close()?

And reset before del_vqs.

> Or is there are way to set up the queues without starting them?
> 
> However I am failing to understand your initial concern, we do reset()
> before del_vqs() in virtbt_remove(). Should we be doing something different
> in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
> keep buffers attached if they are not used.
> 
> Regards
> 
> Marcel

They are not used at that point but until device is reset can use them.
Also, if you then proceed to open without a reset, and kick,
device will start by processing the original buffers, crashing
or corrupting memory.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Nov 25, 2021 at 10:58:56PM +0100, Marcel Holtmann wrote:
> Hi Michael,
> 
> >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> >>>> 
> >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >>>> to be doing. These are transport enable/disable callbacks from the BT
> >>>> Core towards the driver. It maps to a device being enabled/disabled by
> >>>> something like bluetoothd for example. So if disabled, I expect that no
> >>>> resources/queues are in use.
> >>>> 
> >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> >>>> with all other transports like USB, SDIO, UART etc.
> >>>> 
> >>>>> The cost here is a single skb wasted on an unused bt device - which
> >>>>> seems modest.
> >>>> 
> >>>> There should be no buffer used if the device is powered off. We also don’t
> >>>> have any USB URBs in-flight if the transport is not active.
> >>>> 
> >>>>> NB: with this fix in place driver still suffers from a race condition if
> >>>>> an interrupt triggers while device is being reset. Work on a fix for
> >>>>> that issue is in progress.
> >>>> 
> >>>> In the virtbt_close() callback we should deactivate all interrupts.
> >>>> 
> >>> 
> >>> If you want to do that then device has to be reset on close,
> >>> and fully reinitialized on open.
> >>> Can you work on a patch like that?
> >>> Given I don't have the device such a rework is probably more
> >>> than I can undertake.
> >> 
> >> so you mean move virtio_find_vqs() into virtbt_open() and del_vqs() into
> >> virtbt_close()?
> > 
> > And reset before del_vqs.
> > 
> >> Or is there are way to set up the queues without starting them?
> >> 
> >> However I am failing to understand your initial concern, we do reset()
> >> before del_vqs() in virtbt_remove(). Should we be doing something different
> >> in virtbt_close() other than virtqueue_detach_unused_buf(). Why would I
> >> keep buffers attached if they are not used.
> >> 
> > 
> > They are not used at that point but until device is reset can use them.
> > Also, if you then proceed to open without a reset, and kick,
> > device will start by processing the original buffers, crashing
> > or corrupting memory.
> 
> so the only valid usage is like this:
> 
> 	vdev->config->reset(vdev);
> 
> 	while ((.. = virtqueue_detach_unused_buf(vq))) {
> 	}
> 
> 	vdev->config->del_vqs(vdev);
> 
> If I make virtbt_{open,close} a NOP, then I keep adding an extra SKB to inbuf on
> every power cycle (ifup/ifdown).

So make sure you don't :)

> How does netdev handle this?
> 
> Regards
> 
> Marcel

For net, open adds buffers to vq. close does not free them up -
they stay in the vq until device is removed.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> Hi Michael,
> 
> > Device removal is clearly out of virtio spec: it attempts to remove
> > unused buffers from a VQ before invoking device reset. To fix, make
> > open/close NOPs and do all cleanup/setup in probe/remove.
> 
> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> to be doing. These are transport enable/disable callbacks from the BT
> Core towards the driver. It maps to a device being enabled/disabled by
> something like bluetoothd for example. So if disabled, I expect that no
> resources/queues are in use.
> 
> Maybe I misunderstand the virtio spec in that regard, but I would like
> to keep this fundamental concept of a Bluetooth driver. It does work
> with all other transports like USB, SDIO, UART etc.
> 
> > The cost here is a single skb wasted on an unused bt device - which
> > seems modest.
> 
> There should be no buffer used if the device is powered off. We also don’t
> have any USB URBs in-flight if the transport is not active.
> 
> > NB: with this fix in place driver still suffers from a race condition if
> > an interrupt triggers while device is being reset. Work on a fix for
> > that issue is in progress.
> 
> In the virtbt_close() callback we should deactivate all interrupts.
> 
> Regards
> 
> Marcel

So Marcel, do I read it right that you are working on a fix
and I can drop this patch for now?

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Dec 09, 2021 at 04:22:58PM -0500, Michael S. Tsirkin wrote:
> On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> > Hi Michael,
> > 
> > > Device removal is clearly out of virtio spec: it attempts to remove
> > > unused buffers from a VQ before invoking device reset. To fix, make
> > > open/close NOPs and do all cleanup/setup in probe/remove.
> > 
> > so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > to be doing. These are transport enable/disable callbacks from the BT
> > Core towards the driver. It maps to a device being enabled/disabled by
> > something like bluetoothd for example. So if disabled, I expect that no
> > resources/queues are in use.
> > 
> > Maybe I misunderstand the virtio spec in that regard, but I would like
> > to keep this fundamental concept of a Bluetooth driver. It does work
> > with all other transports like USB, SDIO, UART etc.
> > 
> > > The cost here is a single skb wasted on an unused bt device - which
> > > seems modest.
> > 
> > There should be no buffer used if the device is powered off. We also don’t
> > have any USB URBs in-flight if the transport is not active.
> > 
> > > NB: with this fix in place driver still suffers from a race condition if
> > > an interrupt triggers while device is being reset. Work on a fix for
> > > that issue is in progress.
> > 
> > In the virtbt_close() callback we should deactivate all interrupts.
> > 
> > Regards
> > 
> > Marcel
> 
> So Marcel, do I read it right that you are working on a fix
> and I can drop this patch for now?

ping

> -- 
> MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Mon, Dec 13, 2021 at 05:44:13AM -0500, Michael S. Tsirkin wrote:
> On Thu, Dec 09, 2021 at 04:22:58PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Nov 25, 2021 at 09:02:01PM +0100, Marcel Holtmann wrote:
> > > Hi Michael,
> > > 
> > > > Device removal is clearly out of virtio spec: it attempts to remove
> > > > unused buffers from a VQ before invoking device reset. To fix, make
> > > > open/close NOPs and do all cleanup/setup in probe/remove.
> > > 
> > > so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > > to be doing. These are transport enable/disable callbacks from the BT
> > > Core towards the driver. It maps to a device being enabled/disabled by
> > > something like bluetoothd for example. So if disabled, I expect that no
> > > resources/queues are in use.
> > > 
> > > Maybe I misunderstand the virtio spec in that regard, but I would like
> > > to keep this fundamental concept of a Bluetooth driver. It does work
> > > with all other transports like USB, SDIO, UART etc.
> > > 
> > > > The cost here is a single skb wasted on an unused bt device - which
> > > > seems modest.
> > > 
> > > There should be no buffer used if the device is powered off. We also don’t
> > > have any USB URBs in-flight if the transport is not active.
> > > 
> > > > NB: with this fix in place driver still suffers from a race condition if
> > > > an interrupt triggers while device is being reset. Work on a fix for
> > > > that issue is in progress.
> > > 
> > > In the virtbt_close() callback we should deactivate all interrupts.
> > > 
> > > Regards
> > > 
> > > Marcel
> > 
> > So Marcel, do I read it right that you are working on a fix
> > and I can drop this patch for now?
> 
> ping


If I don't hear otherwise I'll queue my version - it might not
be ideal but it at least does not violate the spec.
We can work on not allocating/freeing buffers later
as appropriate.

> > -- 
> > MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Thu, Dec 16, 2021 at 08:58:31PM +0100, Marcel Holtmann wrote:
> Hi Michael,
> 
> >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> >>>> 
> >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> >>>> to be doing. These are transport enable/disable callbacks from the BT
> >>>> Core towards the driver. It maps to a device being enabled/disabled by
> >>>> something like bluetoothd for example. So if disabled, I expect that no
> >>>> resources/queues are in use.
> >>>> 
> >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> >>>> with all other transports like USB, SDIO, UART etc.
> >>>> 
> >>>>> The cost here is a single skb wasted on an unused bt device - which
> >>>>> seems modest.
> >>>> 
> >>>> There should be no buffer used if the device is powered off. We also don’t
> >>>> have any USB URBs in-flight if the transport is not active.
> >>>> 
> >>>>> NB: with this fix in place driver still suffers from a race condition if
> >>>>> an interrupt triggers while device is being reset. Work on a fix for
> >>>>> that issue is in progress.
> >>>> 
> >>>> In the virtbt_close() callback we should deactivate all interrupts.
> >>>> 
> >>>> Regards
> >>>> 
> >>>> Marcel
> >>> 
> >>> So Marcel, do I read it right that you are working on a fix
> >>> and I can drop this patch for now?
> >> 
> >> ping
> > 
> > 
> > If I don't hear otherwise I'll queue my version - it might not
> > be ideal but it at least does not violate the spec.
> > We can work on not allocating/freeing buffers later
> > as appropriate.
> 
> I have a patch, but it is not fully tested yet.
> 
> Regards
> 
> Marcel

ping

it's been a month ...

I'm working on cleaning up module/device removal in virtio and bt
is kind of sticking out.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Fri, Jan 14, 2022 at 03:12:47PM -0500, Michael S. Tsirkin wrote:
> On Thu, Dec 16, 2021 at 08:58:31PM +0100, Marcel Holtmann wrote:
> > Hi Michael,
> > 
> > >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> > >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> > >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> > >>>> 
> > >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > >>>> to be doing. These are transport enable/disable callbacks from the BT
> > >>>> Core towards the driver. It maps to a device being enabled/disabled by
> > >>>> something like bluetoothd for example. So if disabled, I expect that no
> > >>>> resources/queues are in use.
> > >>>> 
> > >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> > >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> > >>>> with all other transports like USB, SDIO, UART etc.
> > >>>> 
> > >>>>> The cost here is a single skb wasted on an unused bt device - which
> > >>>>> seems modest.
> > >>>> 
> > >>>> There should be no buffer used if the device is powered off. We also don’t
> > >>>> have any USB URBs in-flight if the transport is not active.
> > >>>> 
> > >>>>> NB: with this fix in place driver still suffers from a race condition if
> > >>>>> an interrupt triggers while device is being reset. Work on a fix for
> > >>>>> that issue is in progress.
> > >>>> 
> > >>>> In the virtbt_close() callback we should deactivate all interrupts.
> > >>>> 
> > >>>> Regards
> > >>>> 
> > >>>> Marcel
> > >>> 
> > >>> So Marcel, do I read it right that you are working on a fix
> > >>> and I can drop this patch for now?
> > >> 
> > >> ping
> > > 
> > > 
> > > If I don't hear otherwise I'll queue my version - it might not
> > > be ideal but it at least does not violate the spec.
> > > We can work on not allocating/freeing buffers later
> > > as appropriate.
> > 
> > I have a patch, but it is not fully tested yet.
> > 
> > Regards
> > 
> > Marcel
> 
> ping
> 
> it's been a month ...
> 
> I'm working on cleaning up module/device removal in virtio and bt
> is kind of sticking out.

I am inclined to make this driver depend on BROKEN for now.
Any objections?


> -- 
> MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] Bluetooth: virtio_bt: fix device removal

[Michael S. Tsirkin]

On Mon, Jun 13, 2022 at 02:58:59AM -0400, Michael S. Tsirkin wrote:
> On Fri, Jan 14, 2022 at 03:12:47PM -0500, Michael S. Tsirkin wrote:
> > On Thu, Dec 16, 2021 at 08:58:31PM +0100, Marcel Holtmann wrote:
> > > Hi Michael,
> > > 
> > > >>>>> Device removal is clearly out of virtio spec: it attempts to remove
> > > >>>>> unused buffers from a VQ before invoking device reset. To fix, make
> > > >>>>> open/close NOPs and do all cleanup/setup in probe/remove.
> > > >>>> 
> > > >>>> so the virtbt_{open,close} as NOP is not really what a driver is suppose
> > > >>>> to be doing. These are transport enable/disable callbacks from the BT
> > > >>>> Core towards the driver. It maps to a device being enabled/disabled by
> > > >>>> something like bluetoothd for example. So if disabled, I expect that no
> > > >>>> resources/queues are in use.
> > > >>>> 
> > > >>>> Maybe I misunderstand the virtio spec in that regard, but I would like
> > > >>>> to keep this fundamental concept of a Bluetooth driver. It does work
> > > >>>> with all other transports like USB, SDIO, UART etc.
> > > >>>> 
> > > >>>>> The cost here is a single skb wasted on an unused bt device - which
> > > >>>>> seems modest.
> > > >>>> 
> > > >>>> There should be no buffer used if the device is powered off. We also don’t
> > > >>>> have any USB URBs in-flight if the transport is not active.
> > > >>>> 
> > > >>>>> NB: with this fix in place driver still suffers from a race condition if
> > > >>>>> an interrupt triggers while device is being reset. Work on a fix for
> > > >>>>> that issue is in progress.
> > > >>>> 
> > > >>>> In the virtbt_close() callback we should deactivate all interrupts.
> > > >>>> 
> > > >>>> Regards
> > > >>>> 
> > > >>>> Marcel
> > > >>> 
> > > >>> So Marcel, do I read it right that you are working on a fix
> > > >>> and I can drop this patch for now?
> > > >> 
> > > >> ping
> > > > 
> > > > 
> > > > If I don't hear otherwise I'll queue my version - it might not
> > > > be ideal but it at least does not violate the spec.
> > > > We can work on not allocating/freeing buffers later
> > > > as appropriate.
> > > 
> > > I have a patch, but it is not fully tested yet.
> > > 
> > > Regards
> > > 
> > > Marcel
> > 
> > ping
> > 
> > it's been a month ...
> > 
> > I'm working on cleaning up module/device removal in virtio and bt
> > is kind of sticking out.
> 
> I am inclined to make this driver depend on BROKEN for now.
> Any objections?

OK patch incoming.

> 
> > -- 
> > MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH] Bluetooth: virtio_bt: fix device removal

[Igor Skalkin]

Tested-by: Igor Skalkin <igor.skalkin@opensynergy.com>

Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization