[PATCH v3 0/3] vduse: add support for networking devices

[PATCH v3 0/3] vduse: add support for networking devices

[Maxime Coquelin]

This small series enables virtio-net device type in VDUSE.
With it, basic operation have been tested, both with
virtio-vdpa and vhost-vdpa using DPDK Vhost library series
adding VDUSE support using split rings layout (merged in
DPDK v23.07-rc1).

Control queue support (and so multiqueue) has also been
tested, but requires a Kernel series from Jason Wang
relaxing control queue polling [1] to function reliably,
so while Jason rework is done, a patch is added to disable
CVQ and features that depend on it (tested also with DPDK
v23.07-rc1).

[1]: https://lore.kernel.org/lkml/CACGkMEtgrxN3PPwsDo4oOsnsSLJfEmBEZ0WvjGRr3whU+QasUg@mail.gmail.com/T/

v2 -> v3 changes:
=================
- Use allow list instead of deny list (Michael)

v1 -> v2 changes:
=================
- Add a patch to disable CVQ (Michael)

RFC -> v1 changes:
==================
- Fail device init if it does not support VERSION_1 (Jason)

Maxime Coquelin (3):
  vduse: validate block features only with block devices
  vduse: enable Virtio-net device type
  vduse: Temporarily disable control queue features

 drivers/vdpa/vdpa_user/vduse_dev.c | 51 +++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 4 deletions(-)

-- 
2.41.0

[PATCH v3 1/3] vduse: validate block features only with block devices

[Maxime Coquelin]

This patch is preliminary work to enable network device
type support to VDUSE.

As VIRTIO_BLK_F_CONFIG_WCE shares the same value as
VIRTIO_NET_F_HOST_TSO4, we need to restrict its check
to Virtio-blk device type.

Acked-by: Jason Wang <jasowang@redhat.com>
Reviewed-by: Xie Yongji <xieyongji@bytedance.com>
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index dc38ed21319d..ff9fdd6783fe 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -1662,13 +1662,14 @@ static bool device_is_allowed(u32 device_id)
 	return false;
 }
 
-static bool features_is_valid(u64 features)
+static bool features_is_valid(struct vduse_dev_config *config)
 {
-	if (!(features & (1ULL << VIRTIO_F_ACCESS_PLATFORM)))
+	if (!(config->features & (1ULL << VIRTIO_F_ACCESS_PLATFORM)))
 		return false;
 
 	/* Now we only support read-only configuration space */
-	if (features & (1ULL << VIRTIO_BLK_F_CONFIG_WCE))
+	if ((config->device_id == VIRTIO_ID_BLOCK) &&
+			(config->features & (1ULL << VIRTIO_BLK_F_CONFIG_WCE)))
 		return false;
 
 	return true;
@@ -1695,7 +1696,7 @@ static bool vduse_validate_config(struct vduse_dev_config *config)
 	if (!device_is_allowed(config->device_id))
 		return false;
 
-	if (!features_is_valid(config->features))
+	if (!features_is_valid(config))
 		return false;
 
 	return true;
-- 
2.41.0

[PATCH v3 2/3] vduse: enable Virtio-net device type

[Maxime Coquelin]

This patch adds Virtio-net device type to the supported
devices types. Initialization fails if the device does
not support VIRTIO_F_VERSION_1 feature, in order to
guarantee the configuration space is read-only.

Acked-by: Jason Wang <jasowang@redhat.com>
Reviewed-by: Xie Yongji <xieyongji@bytedance.com>
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index ff9fdd6783fe..1271c9796517 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -142,6 +142,7 @@ static struct workqueue_struct *vduse_irq_bound_wq;
 
 static u32 allowed_device_id[] = {
 	VIRTIO_ID_BLOCK,
+	VIRTIO_ID_NET,
 };
 
 static inline struct vduse_dev *vdpa_to_vduse(struct vdpa_device *vdpa)
@@ -1672,6 +1673,10 @@ static bool features_is_valid(struct vduse_dev_config *config)
 			(config->features & (1ULL << VIRTIO_BLK_F_CONFIG_WCE)))
 		return false;
 
+	if ((config->device_id == VIRTIO_ID_NET) &&
+			!(config->features & (1ULL << VIRTIO_F_VERSION_1)))
+		return false;
+
 	return true;
 }
 
@@ -2027,6 +2032,7 @@ static const struct vdpa_mgmtdev_ops vdpa_dev_mgmtdev_ops = {
 
 static struct virtio_device_id id_table[] = {
 	{ VIRTIO_ID_BLOCK, VIRTIO_DEV_ANY_ID },
+	{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
 	{ 0 },
 };
 
-- 
2.41.0

[PATCH v3 3/3] vduse: Temporarily disable control queue features

[Maxime Coquelin]

Virtio-net driver control queue implementation is not safe
when used with VDUSE. If the VDUSE application does not
reply to control queue messages, it currently ends up
hanging the kernel thread sending this command.

Some work is on-going to make the control queue
implementation robust with VDUSE. Until it is completed,
let's filter out control virtqueue and features that depend
on it by keeping only features known to be supported.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 36 ++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 1271c9796517..7345071db0a8 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -46,6 +46,30 @@
 
 #define IRQ_UNBOUND -1
 
+#define VDUSE_NET_VALID_FEATURES_MASK           \
+	(BIT_ULL(VIRTIO_NET_F_CSUM) |           \
+	 BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |     \
+	 BIT_ULL(VIRTIO_NET_F_MTU) |            \
+	 BIT_ULL(VIRTIO_NET_F_MAC) |            \
+	 BIT_ULL(VIRTIO_NET_F_GUEST_TSO4) |     \
+	 BIT_ULL(VIRTIO_NET_F_GUEST_TSO6) |     \
+	 BIT_ULL(VIRTIO_NET_F_GUEST_ECN) |      \
+	 BIT_ULL(VIRTIO_NET_F_GUEST_UFO) |      \
+	 BIT_ULL(VIRTIO_NET_F_HOST_TSO4) |      \
+	 BIT_ULL(VIRTIO_NET_F_HOST_TSO6) |      \
+	 BIT_ULL(VIRTIO_NET_F_HOST_ECN) |       \
+	 BIT_ULL(VIRTIO_NET_F_HOST_UFO) |       \
+	 BIT_ULL(VIRTIO_NET_F_MRG_RXBUF) |      \
+	 BIT_ULL(VIRTIO_NET_F_STATUS) |         \
+	 BIT_ULL(VIRTIO_NET_F_HOST_USO) |       \
+	 BIT_ULL(VIRTIO_F_ANY_LAYOUT) |         \
+	 BIT_ULL(VIRTIO_RING_F_INDIRECT_DESC) | \
+	 BIT_ULL(VIRTIO_F_EVENT_IDX) |          \
+	 BIT_ULL(VIRTIO_F_VERSION_1) |          \
+	 BIT_ULL(VIRTIO_F_IOMMU_PLATFORM) |     \
+	 BIT_ULL(VIRTIO_F_RING_PACKED) |        \
+	 BIT_ULL(VIRTIO_F_IN_ORDER))
+
 struct vduse_virtqueue {
 	u16 index;
 	u16 num_max;
@@ -1778,6 +1802,16 @@ static struct attribute *vduse_dev_attrs[] = {
 
 ATTRIBUTE_GROUPS(vduse_dev);
 
+static void vduse_dev_features_filter(struct vduse_dev_config *config)
+{
+	/*
+	 * Temporarily filter out virtio-net's control virtqueue and features
+	 * that depend on it while CVQ is being made more robust for VDUSE.
+	 */
+	if (config->device_id == VIRTIO_ID_NET)
+		config->features &= VDUSE_NET_VALID_FEATURES_MASK;
+}
+
 static int vduse_create_dev(struct vduse_dev_config *config,
 			    void *config_buf, u64 api_version)
 {
@@ -1793,6 +1827,8 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	if (!dev)
 		goto err;
 
+	vduse_dev_features_filter(config);
+
 	dev->api_version = api_version;
 	dev->device_features = config->features;
 	dev->device_id = config->device_id;
-- 
2.41.0

Re: [PATCH v3 3/3] vduse: Temporarily disable control queue features

[Jason Wang]

On Wed, Jul 5, 2023 at 6:04 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> Virtio-net driver control queue implementation is not safe
> when used with VDUSE. If the VDUSE application does not
> reply to control queue messages, it currently ends up
> hanging the kernel thread sending this command.
>
> Some work is on-going to make the control queue
> implementation robust with VDUSE. Until it is completed,
> let's filter out control virtqueue and features that depend
> on it by keeping only features known to be supported.
>
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Acked-by: Jason Wang <jasowang@redhat.com>

Thanks

> ---
>  drivers/vdpa/vdpa_user/vduse_dev.c | 36 ++++++++++++++++++++++++++++++
>  1 file changed, 36 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> index 1271c9796517..7345071db0a8 100644
> --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> @@ -46,6 +46,30 @@
>
>  #define IRQ_UNBOUND -1
>
> +#define VDUSE_NET_VALID_FEATURES_MASK           \
> +       (BIT_ULL(VIRTIO_NET_F_CSUM) |           \
> +        BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |     \
> +        BIT_ULL(VIRTIO_NET_F_MTU) |            \
> +        BIT_ULL(VIRTIO_NET_F_MAC) |            \
> +        BIT_ULL(VIRTIO_NET_F_GUEST_TSO4) |     \
> +        BIT_ULL(VIRTIO_NET_F_GUEST_TSO6) |     \
> +        BIT_ULL(VIRTIO_NET_F_GUEST_ECN) |      \
> +        BIT_ULL(VIRTIO_NET_F_GUEST_UFO) |      \
> +        BIT_ULL(VIRTIO_NET_F_HOST_TSO4) |      \
> +        BIT_ULL(VIRTIO_NET_F_HOST_TSO6) |      \
> +        BIT_ULL(VIRTIO_NET_F_HOST_ECN) |       \
> +        BIT_ULL(VIRTIO_NET_F_HOST_UFO) |       \
> +        BIT_ULL(VIRTIO_NET_F_MRG_RXBUF) |      \
> +        BIT_ULL(VIRTIO_NET_F_STATUS) |         \
> +        BIT_ULL(VIRTIO_NET_F_HOST_USO) |       \
> +        BIT_ULL(VIRTIO_F_ANY_LAYOUT) |         \
> +        BIT_ULL(VIRTIO_RING_F_INDIRECT_DESC) | \
> +        BIT_ULL(VIRTIO_F_EVENT_IDX) |          \
> +        BIT_ULL(VIRTIO_F_VERSION_1) |          \
> +        BIT_ULL(VIRTIO_F_IOMMU_PLATFORM) |     \
> +        BIT_ULL(VIRTIO_F_RING_PACKED) |        \
> +        BIT_ULL(VIRTIO_F_IN_ORDER))
> +
>  struct vduse_virtqueue {
>         u16 index;
>         u16 num_max;
> @@ -1778,6 +1802,16 @@ static struct attribute *vduse_dev_attrs[] = {
>
>  ATTRIBUTE_GROUPS(vduse_dev);
>
> +static void vduse_dev_features_filter(struct vduse_dev_config *config)
> +{
> +       /*
> +        * Temporarily filter out virtio-net's control virtqueue and features
> +        * that depend on it while CVQ is being made more robust for VDUSE.
> +        */
> +       if (config->device_id == VIRTIO_ID_NET)
> +               config->features &= VDUSE_NET_VALID_FEATURES_MASK;
> +}
> +
>  static int vduse_create_dev(struct vduse_dev_config *config,
>                             void *config_buf, u64 api_version)
>  {
> @@ -1793,6 +1827,8 @@ static int vduse_create_dev(struct vduse_dev_config *config,
>         if (!dev)
>                 goto err;
>
> +       vduse_dev_features_filter(config);
> +
>         dev->api_version = api_version;
>         dev->device_features = config->features;
>         dev->device_id = config->device_id;
> --
> 2.41.0
>

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Michael S. Tsirkin]

On Wed, Jul 05, 2023 at 12:04:27PM +0200, Maxime Coquelin wrote:
> This small series enables virtio-net device type in VDUSE.
> With it, basic operation have been tested, both with
> virtio-vdpa and vhost-vdpa using DPDK Vhost library series
> adding VDUSE support using split rings layout (merged in
> DPDK v23.07-rc1).
> 
> Control queue support (and so multiqueue) has also been
> tested, but requires a Kernel series from Jason Wang
> relaxing control queue polling [1] to function reliably,
> so while Jason rework is done, a patch is added to disable
> CVQ and features that depend on it (tested also with DPDK
> v23.07-rc1).


So I can put this in next, the issue I think is
that of security: currently selinux can if necessary block
access to creating virtio block devices.
But if we have more than one type we need a way for selinux to
block specific types. Can be a patch on top but pls work to
address.

Another question is that with this userspace can inject
packets directly into net stack. Should we check CAP_NET_ADMIN
or such?



> [1]: https://lore.kernel.org/lkml/CACGkMEtgrxN3PPwsDo4oOsnsSLJfEmBEZ0WvjGRr3whU+QasUg@mail.gmail.com/T/
> 
> v2 -> v3 changes:
> =================
> - Use allow list instead of deny list (Michael)
> 
> v1 -> v2 changes:
> =================
> - Add a patch to disable CVQ (Michael)
> 
> RFC -> v1 changes:
> ==================
> - Fail device init if it does not support VERSION_1 (Jason)
> 
> Maxime Coquelin (3):
>   vduse: validate block features only with block devices
>   vduse: enable Virtio-net device type
>   vduse: Temporarily disable control queue features
> 
>  drivers/vdpa/vdpa_user/vduse_dev.c | 51 +++++++++++++++++++++++++++---
>  1 file changed, 47 insertions(+), 4 deletions(-)
> 
> -- 
> 2.41.0

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Jakub Kicinski]

On Thu, 10 Aug 2023 15:04:27 -0400 Michael S. Tsirkin wrote:
> Another question is that with this userspace can inject
> packets directly into net stack. Should we check CAP_NET_ADMIN
> or such?

Directly into the stack? I thought VDUSE is vDPA in user space,
meaning to get to the kernel the packet has to first go thru 
a virtio-net instance.

Or you mean directly into the network?

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Michael S. Tsirkin]

On Thu, Aug 10, 2023 at 02:29:49PM -0700, Jakub Kicinski wrote:
> On Thu, 10 Aug 2023 15:04:27 -0400 Michael S. Tsirkin wrote:
> > Another question is that with this userspace can inject
> > packets directly into net stack. Should we check CAP_NET_ADMIN
> > or such?
> 
> Directly into the stack? I thought VDUSE is vDPA in user space,
> meaning to get to the kernel the packet has to first go thru 
> a virtio-net instance.

yes. is that a sufficient filter in your opinion?

> Or you mean directly into the network?

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Jakub Kicinski]

On Thu, 10 Aug 2023 17:42:11 -0400 Michael S. Tsirkin wrote:
> > Directly into the stack? I thought VDUSE is vDPA in user space,
> > meaning to get to the kernel the packet has to first go thru 
> > a virtio-net instance.  
> 
> yes. is that a sufficient filter in your opinion?

Yes, the ability to create the device feels stronger than CAP_NET_RAW,
and a bit tangential to CAP_NET_ADMIN. But I don't have much practical
experience with virt so no strong opinion, perhaps it does make sense
for someone's deployment? Dunno..

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Maxime Coquelin]

On 8/11/23 00:00, Jakub Kicinski wrote:
> On Thu, 10 Aug 2023 17:42:11 -0400 Michael S. Tsirkin wrote:
>>> Directly into the stack? I thought VDUSE is vDPA in user space,
>>> meaning to get to the kernel the packet has to first go thru
>>> a virtio-net instance.
>>
>> yes. is that a sufficient filter in your opinion?
> 
> Yes, the ability to create the device feels stronger than CAP_NET_RAW,
> and a bit tangential to CAP_NET_ADMIN. But I don't have much practical
> experience with virt so no strong opinion, perhaps it does make sense
> for someone's deployment? Dunno..
> 

I'm not sure CAP_NET_ADMIN should be required for creating the VDUSE
devices, as the device could be attached to vhost-vDPA and so not
visible to the Kernel networking stack.

However, CAP_NET_ADMIN should be required to attach the VDUSE device to 
virtio-vdpa/virtio-net.

Does that make sense?

Maxime

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Michael S. Tsirkin]

On Tue, Aug 29, 2023 at 03:34:06PM +0200, Maxime Coquelin wrote:
> 
> 
> On 8/11/23 00:00, Jakub Kicinski wrote:
> > On Thu, 10 Aug 2023 17:42:11 -0400 Michael S. Tsirkin wrote:
> > > > Directly into the stack? I thought VDUSE is vDPA in user space,
> > > > meaning to get to the kernel the packet has to first go thru
> > > > a virtio-net instance.
> > > 
> > > yes. is that a sufficient filter in your opinion?
> > 
> > Yes, the ability to create the device feels stronger than CAP_NET_RAW,
> > and a bit tangential to CAP_NET_ADMIN. But I don't have much practical
> > experience with virt so no strong opinion, perhaps it does make sense
> > for someone's deployment? Dunno..
> > 
> 
> I'm not sure CAP_NET_ADMIN should be required for creating the VDUSE
> devices, as the device could be attached to vhost-vDPA and so not
> visible to the Kernel networking stack.
> 
> However, CAP_NET_ADMIN should be required to attach the VDUSE device to
> virtio-vdpa/virtio-net.
> 
> Does that make sense?
> 
> Maxime

OK. How are we going to enforce it?
Also, we need a way for selinux to enable/disable some of these things
but not others.

-- 
MST

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Maxime Coquelin]

On 8/29/23 19:05, Michael S. Tsirkin wrote:
> On Tue, Aug 29, 2023 at 03:34:06PM +0200, Maxime Coquelin wrote:
>>
>>
>> On 8/11/23 00:00, Jakub Kicinski wrote:
>>> On Thu, 10 Aug 2023 17:42:11 -0400 Michael S. Tsirkin wrote:
>>>>> Directly into the stack? I thought VDUSE is vDPA in user space,
>>>>> meaning to get to the kernel the packet has to first go thru
>>>>> a virtio-net instance.
>>>>
>>>> yes. is that a sufficient filter in your opinion?
>>>
>>> Yes, the ability to create the device feels stronger than CAP_NET_RAW,
>>> and a bit tangential to CAP_NET_ADMIN. But I don't have much practical
>>> experience with virt so no strong opinion, perhaps it does make sense
>>> for someone's deployment? Dunno..
>>>
>>
>> I'm not sure CAP_NET_ADMIN should be required for creating the VDUSE
>> devices, as the device could be attached to vhost-vDPA and so not
>> visible to the Kernel networking stack.
>>
>> However, CAP_NET_ADMIN should be required to attach the VDUSE device to
>> virtio-vdpa/virtio-net.
>>
>> Does that make sense?
>>
>> Maxime
> 
> OK. How are we going to enforce it?

Actually, it seems already enforced for all VDPA devices types.
Indeed, the VDPA_CMD_DEV_NEW Netlink command used to add the device to
the VDPA bus has the GENL_ADMIN_PERM flag set, and so require
CAT_NET_ADMIN.

> Also, we need a way for selinux to enable/disable some of these things
> but not others.

Ok, I can do it in a patch on top.
Do you have a pointer where it is done for Virtio Block devices?

Maxime

Re: [PATCH v3 0/3] vduse: add support for networking devices

[Michael S. Tsirkin]

On Wed, Aug 30, 2023 at 01:27:18PM +0200, Maxime Coquelin wrote:
> 
> 
> On 8/29/23 19:05, Michael S. Tsirkin wrote:
> > On Tue, Aug 29, 2023 at 03:34:06PM +0200, Maxime Coquelin wrote:
> > > 
> > > 
> > > On 8/11/23 00:00, Jakub Kicinski wrote:
> > > > On Thu, 10 Aug 2023 17:42:11 -0400 Michael S. Tsirkin wrote:
> > > > > > Directly into the stack? I thought VDUSE is vDPA in user space,
> > > > > > meaning to get to the kernel the packet has to first go thru
> > > > > > a virtio-net instance.
> > > > > 
> > > > > yes. is that a sufficient filter in your opinion?
> > > > 
> > > > Yes, the ability to create the device feels stronger than CAP_NET_RAW,
> > > > and a bit tangential to CAP_NET_ADMIN. But I don't have much practical
> > > > experience with virt so no strong opinion, perhaps it does make sense
> > > > for someone's deployment? Dunno..
> > > > 
> > > 
> > > I'm not sure CAP_NET_ADMIN should be required for creating the VDUSE
> > > devices, as the device could be attached to vhost-vDPA and so not
> > > visible to the Kernel networking stack.
> > > 
> > > However, CAP_NET_ADMIN should be required to attach the VDUSE device to
> > > virtio-vdpa/virtio-net.
> > > 
> > > Does that make sense?
> > > 
> > > Maxime
> > 
> > OK. How are we going to enforce it?
> 
> Actually, it seems already enforced for all VDPA devices types.
> Indeed, the VDPA_CMD_DEV_NEW Netlink command used to add the device to
> the VDPA bus has the GENL_ADMIN_PERM flag set, and so require
> CAT_NET_ADMIN.

Hmm good point. Pity I didn't notice earlier. Oh well there's always
the next release.

> > Also, we need a way for selinux to enable/disable some of these things
> > but not others.
> 
> Ok, I can do it in a patch on top.
> Do you have a pointer where it is done for Virtio Block devices?
> 
> Maxime

It's not done yet - at the moment vduse device is always block so we
didn't need the distinction.

-- 
MST