From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C5C3C43334 for ; Wed, 22 Jun 2022 10:22:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236201AbiFVKWN (ORCPT ); Wed, 22 Jun 2022 06:22:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232489AbiFVKWM (ORCPT ); Wed, 22 Jun 2022 06:22:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2120E3AA60 for ; Wed, 22 Jun 2022 03:22:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655893330; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=nfl1nb+33mlRf2GSwN8w3YvpZV0ySkPvRLOSfPPc+dc=; b=VbwuZnG9oa5oa0by2s7ltzHtGqg2DVtKhB5Ca5OI1dY08HQfGJzMkv5Aq0mXmeaovdHy6p r3mF4yQTaPRfKTtdckCkem5NYu1wRTwGksZCVTbsJPXFguGUT5e+kIXy6QkgOu4z8enUip bBuTNkbhkgS6ouJ8WEE668JyZ+K7IRM= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-550-eTpCEBYZPE2absbPUAZLIg-1; Wed, 22 Jun 2022 06:22:08 -0400 X-MC-Unique: eTpCEBYZPE2absbPUAZLIg-1 Received: by mail-qk1-f199.google.com with SMTP id i10-20020a05620a404a00b006a7609f54c6so19425791qko.7 for ; Wed, 22 Jun 2022 03:22:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=nfl1nb+33mlRf2GSwN8w3YvpZV0ySkPvRLOSfPPc+dc=; b=YBH7bsuppwJkB994Y5QvIHKucw5U4a5CFJzmdEE8fSeVMaYJ3iJsqVWLaAhNOWodKM Cx6QNAH/xHUTrFCaHVXaDjDNtM0Ea43x32NeF8k+buKQGCWFgwVzhfmLDgLfC+SPT1nh uw259IFvvQDxLutAvFpaOk5tnBp6fHUqSDQiRXpcuGy7NPqq+hgtLiQvI9SauX8wlxCY qHqhD6f1qexSx3t5AeDDsy8hJE2evSAE/iF2MiqBuRvbj/Kqi9V5C8VzLjMOImaZRFMc msOHJc1vbbI9FLV6qKpKgmbW/za3kC9NcByn6pmojUaDBXuRFISHvYQ/kltOgzuh3gEe WzUw== X-Gm-Message-State: AJIora9xZkcqTXotUNG8zjdW6KJApE2qGSsXQrtTQGEQb69c+UY0Nc1C 23XPizMHpk3JBLjb2dX0cnV0a0carsoMCyU7Q7d6JGd2dYqJNFkIDpvW4zyfQDDmB6l+Cio3pFC BNaPLF4w04L3S7HX3ZA9imeeVfrQWDUdL X-Received: by 2002:a05:6214:1c83:b0:46b:a79a:2f0b with SMTP id ib3-20020a0562141c8300b0046ba79a2f0bmr26575110qvb.103.1655893328396; Wed, 22 Jun 2022 03:22:08 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tRH9IgjI7D7T9+pD0EtLZxv84aRvOyAbEbYmJ/axV5leG89dyYdugg4VxhD4kTKIAEuACDpoqbEz6mSVCQhYE= X-Received: by 2002:a05:6214:1c83:b0:46b:a79a:2f0b with SMTP id ib3-20020a0562141c8300b0046ba79a2f0bmr26575071qvb.103.1655893328078; Wed, 22 Jun 2022 03:22:08 -0700 (PDT) MIME-Version: 1.0 References: <20220330180436.24644-1-gdawar@xilinx.com> <20220330180436.24644-20-gdawar@xilinx.com> In-Reply-To: From: Eugenio Perez Martin Date: Wed, 22 Jun 2022 12:21:32 +0200 Message-ID: Subject: Re: [PATCH v2 19/19] vdpasim: control virtqueue support To: Stefano Garzarella Cc: Gautam Dawar , Jason Wang , "Michael S. Tsirkin" , Xie Yongji , Gautam Dawar , Longpeng , Eli Cohen , Parav Pandit , Linux Virtualization , kernel list , kvm , netdev , Martin Petrus Hubertus Habets , Harpreet Singh Anand , Martin Porter , Pablo Cascon Katchadourian , Dinan Gunawardena , "Kamde, Tanuj" , habetsm.xilinx@gmail.com, ecree.xilinx@gmail.com, Wu Zongyong , Christophe JAILLET , Zhu Lingshan , Si-Wei Liu , Dan Carpenter , Zhang Min Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Tue, Jun 21, 2022 at 5:20 PM Stefano Garzarella wrote: > > Hi Gautam, > > On Wed, Mar 30, 2022 at 8:21 PM Gautam Dawar wrote: > > > > This patch introduces the control virtqueue support for vDPA > > simulator. This is a requirement for supporting advanced features like > > multiqueue. > > > > A requirement for control virtqueue is to isolate its memory access > > from the rx/tx virtqueues. This is because when using vDPA device > > for VM, the control virqueue is not directly assigned to VM. Userspace > > (Qemu) will present a shadow control virtqueue to control for > > recording the device states. > > > > The isolation is done via the virtqueue groups and ASID support in > > vDPA through vhost-vdpa. The simulator is extended to have: > > > > 1) three virtqueues: RXVQ, TXVQ and CVQ (control virtqueue) > > 2) two virtqueue groups: group 0 contains RXVQ and TXVQ; group 1 > > contains CVQ > > 3) two address spaces and the simulator simply implements the address > > spaces by mapping it 1:1 to IOTLB. > > > > For the VM use cases, userspace(Qemu) may set AS 0 to group 0 and AS 1 > > to group 1. So we have: > > > > 1) The IOTLB for virtqueue group 0 contains the mappings of guest, so > > RX and TX can be assigned to guest directly. > > 2) The IOTLB for virtqueue group 1 contains the mappings of CVQ which > > is the buffers that allocated and managed by VMM only. So CVQ of > > vhost-vdpa is visible to VMM only. And Guest can not access the CVQ > > of vhost-vdpa. > > > > For the other use cases, since AS 0 is associated to all virtqueue > > groups by default. All virtqueues share the same mapping by default. > > > > To demonstrate the function, VIRITO_NET_F_CTRL_MACADDR is > > implemented in the simulator for the driver to set mac address. > > > > Signed-off-by: Jason Wang > > Signed-off-by: Gautam Dawar > > --- > > drivers/vdpa/vdpa_sim/vdpa_sim.c | 91 ++++++++++++++++++++++------ > > drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + > > drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 88 ++++++++++++++++++++++++++- > > 3 files changed, 161 insertions(+), 20 deletions(-) > > > > diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c > > index 659e2e2e4b0c..51bd0bafce06 100644 > > --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c > > +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c > > @@ -96,11 +96,17 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim) > > { > > int i; > > > > - for (i = 0; i < vdpasim->dev_attr.nvqs; i++) > > + spin_lock(&vdpasim->iommu_lock); > > + > > + for (i = 0; i < vdpasim->dev_attr.nvqs; i++) { > > vdpasim_vq_reset(vdpasim, &vdpasim->vqs[i]); > > + vringh_set_iotlb(&vdpasim->vqs[i].vring, &vdpasim->iommu[0], > > + &vdpasim->iommu_lock); > > + } > > + > > + for (i = 0; i < vdpasim->dev_attr.nas; i++) > > + vhost_iotlb_reset(&vdpasim->iommu[i]); > > > > - spin_lock(&vdpasim->iommu_lock); > > - vhost_iotlb_reset(vdpasim->iommu); > > spin_unlock(&vdpasim->iommu_lock); > > > > vdpasim->features = 0; > > @@ -145,7 +151,7 @@ static dma_addr_t vdpasim_map_range(struct vdpasim *vdpasim, phys_addr_t paddr, > > dma_addr = iova_dma_addr(&vdpasim->iova, iova); > > > > spin_lock(&vdpasim->iommu_lock); > > - ret = vhost_iotlb_add_range(vdpasim->iommu, (u64)dma_addr, > > + ret = vhost_iotlb_add_range(&vdpasim->iommu[0], (u64)dma_addr, > > (u64)dma_addr + size - 1, (u64)paddr, perm); > > spin_unlock(&vdpasim->iommu_lock); > > > > @@ -161,7 +167,7 @@ static void vdpasim_unmap_range(struct vdpasim *vdpasim, dma_addr_t dma_addr, > > size_t size) > > { > > spin_lock(&vdpasim->iommu_lock); > > - vhost_iotlb_del_range(vdpasim->iommu, (u64)dma_addr, > > + vhost_iotlb_del_range(&vdpasim->iommu[0], (u64)dma_addr, > > (u64)dma_addr + size - 1); > > spin_unlock(&vdpasim->iommu_lock); > > > > @@ -250,8 +256,9 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) > > else > > ops = &vdpasim_config_ops; > > > > - vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, 1, > > - 1, dev_attr->name, false); > > + vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, > > + dev_attr->ngroups, dev_attr->nas, > > + dev_attr->name, false); > > if (IS_ERR(vdpasim)) { > > ret = PTR_ERR(vdpasim); > > goto err_alloc; > > @@ -278,16 +285,20 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) > > if (!vdpasim->vqs) > > goto err_iommu; > > > > - vdpasim->iommu = vhost_iotlb_alloc(max_iotlb_entries, 0); > > + vdpasim->iommu = kmalloc_array(vdpasim->dev_attr.nas, > > + sizeof(*vdpasim->iommu), GFP_KERNEL); > > if (!vdpasim->iommu) > > goto err_iommu; > > > > + for (i = 0; i < vdpasim->dev_attr.nas; i++) > > + vhost_iotlb_init(&vdpasim->iommu[i], 0, 0); > > + > > vdpasim->buffer = kvmalloc(dev_attr->buffer_size, GFP_KERNEL); > > if (!vdpasim->buffer) > > goto err_iommu; > > > > for (i = 0; i < dev_attr->nvqs; i++) > > - vringh_set_iotlb(&vdpasim->vqs[i].vring, vdpasim->iommu, > > + vringh_set_iotlb(&vdpasim->vqs[i].vring, &vdpasim->iommu[0], > > &vdpasim->iommu_lock); > > > > ret = iova_cache_get(); > > @@ -401,7 +412,11 @@ static u32 vdpasim_get_vq_align(struct vdpa_device *vdpa) > > > > static u32 vdpasim_get_vq_group(struct vdpa_device *vdpa, u16 idx) > > { > > - return 0; > > + /* RX and TX belongs to group 0, CVQ belongs to group 1 */ > > + if (idx == 2) > > + return 1; > > + else > > + return 0; > > This code only works for the vDPA-net simulator, since > vdpasim_get_vq_group() is also shared with other simulators (e.g. > vdpa_sim_blk), That's totally right. > should we move this net-specific code into > vdpa_sim_net.c, maybe adding a callback implemented by the different > simulators? > At this moment, VDPASIM_BLK_VQ_NUM is fixed to 1, so maybe the right thing to do for the -rc phase is to check if idx > vdpasim.attr.nvqs? It's a more general fix. For the general case, yes, a callback should be issued to the actual simulator so it's not a surprise when VDPASIM_BLK_VQ_NUM increases, either dynamically or by anyone testing it. Thoughts? Thanks!