From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Wang Subject: Re: [virtio-dev] Re: [PATCH v11 6/6] virtio-balloon: VIRTIO_BALLOON_F_CMD_VQ Date: Thu, 22 Jun 2017 16:40:39 +0800 Message-ID: <594B8287.6000706@intel.com> References: <1497004901-30593-1-git-send-email-wei.w.wang@intel.com> <1497004901-30593-7-git-send-email-wei.w.wang@intel.com> <20170620190343-mutt-send-email-mst@kernel.org> <5949E7C0.3050106@intel.com> <20170621151922-mutt-send-email-mst@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "david@redhat.com" , "Hansen, Dave" , "cornelia.huck@de.ibm.com" , "akpm@linux-foundation.org" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "liliang.opensource@gmail.co To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20170621151922-mutt-send-email-mst@kernel.org> Sender: linux-kernel-owner@vger.kernel.org List-Id: kvm.vger.kernel.org On 06/21/2017 08:28 PM, Michael S. Tsirkin wrote: > On Wed, Jun 21, 2017 at 11:28:00AM +0800, Wei Wang wrote: >> On 06/21/2017 12:18 AM, Michael S. Tsirkin wrote: >>> On Fri, Jun 09, 2017 at 06:41:41PM +0800, Wei Wang wrote: >>>> - if (!virtqueue_indirect_desc_table_add(vq, desc, num)) { >>>> + if (!virtqueue_indirect_desc_table_add(vq, desc, *num)) { >>>> virtqueue_kick(vq); >>>> - wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>>> - vb->balloon_page_chunk.chunk_num = 0; >>>> + if (busy_wait) >>>> + while (!virtqueue_get_buf(vq, &len) && >>>> + !virtqueue_is_broken(vq)) >>>> + cpu_relax(); >>>> + else >>>> + wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>> This is something I didn't previously notice. >>> As you always keep a single buffer in flight, you do not >>> really need indirect at all. Just add all descriptors >>> in the ring directly, then kick. >>> >>> E.g. >>> virtqueue_add_first >>> virtqueue_add_next >>> virtqueue_add_last >>> >>> ? >>> >>> You also want a flag to avoid allocations but there's no need to do it >>> per descriptor, set it on vq. >>> >> Without using the indirect table, I'm thinking about changing to use >> the standard sg (i.e. struct scatterlist), instead of vring_desc, so that >> we don't need to modify or add any new functions of virtqueue_add(). >> >> In this case, we will kmalloc an array of sgs in probe(), and we can add >> the sgs one by one to the vq, which won't trigger the allocation of an >> indirect table inside virtqueue_add(), and then kick when all are added. >> >> Best, >> Wei > And allocate headers too? This can work. API extensions aren't > necessarily a bad idea though. The API I suggest above is preferable > for the simple reason that it can work without INDIRECT flag > support in hypervisor. OK, probably we don't need to add a desc to the vq - we can just use the vq's desc, like this: int virtqueue_add_first(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, unsigned int *idx) { ... uint16_t desc_flags = in ? VRING_DESC_F_NEXT | VRING_DESC_F_WRITE : VRING_DESC_F_NEXT; vq->vring.desc[vq->free_head].addr = addr; vq->vring.desc[vq->free_head].len = len; vq->vring.desc[vq->free_head].flags = cpu_to_virtio16(_vq->vdev, flags); /* return to the caller the desc id */ *idx = vq->free_head; ... } int virtqueue_add_next(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, bool end, unsigned int *idx) { ... vq->vring.desc[*idx].next = vq->free_head; vq->vring.desc[vq->free_head].addr = addr; ... if (end) remove the VRING_DESC_F_NEXT flag } What do you think? We can also combine the two functions into one. Best, Wei From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f200.google.com (mail-pf0-f200.google.com [209.85.192.200]) by kanga.kvack.org (Postfix) with ESMTP id 124A16B0279 for ; Thu, 22 Jun 2017 04:38:21 -0400 (EDT) Received: by mail-pf0-f200.google.com with SMTP id d62so8880227pfb.13 for ; Thu, 22 Jun 2017 01:38:21 -0700 (PDT) Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id u1si757700plj.51.2017.06.22.01.38.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Jun 2017 01:38:20 -0700 (PDT) Message-ID: <594B8287.6000706@intel.com> Date: Thu, 22 Jun 2017 16:40:39 +0800 From: Wei Wang MIME-Version: 1.0 Subject: Re: [virtio-dev] Re: [PATCH v11 6/6] virtio-balloon: VIRTIO_BALLOON_F_CMD_VQ References: <1497004901-30593-1-git-send-email-wei.w.wang@intel.com> <1497004901-30593-7-git-send-email-wei.w.wang@intel.com> <20170620190343-mutt-send-email-mst@kernel.org> <5949E7C0.3050106@intel.com> <20170621151922-mutt-send-email-mst@kernel.org> In-Reply-To: <20170621151922-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: "Michael S. Tsirkin" Cc: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "david@redhat.com" , "Hansen, Dave" , "cornelia.huck@de.ibm.com" , "akpm@linux-foundation.org" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "liliang.opensource@gmail.com" , "riel@redhat.com" , "nilal@redhat.com" On 06/21/2017 08:28 PM, Michael S. Tsirkin wrote: > On Wed, Jun 21, 2017 at 11:28:00AM +0800, Wei Wang wrote: >> On 06/21/2017 12:18 AM, Michael S. Tsirkin wrote: >>> On Fri, Jun 09, 2017 at 06:41:41PM +0800, Wei Wang wrote: >>>> - if (!virtqueue_indirect_desc_table_add(vq, desc, num)) { >>>> + if (!virtqueue_indirect_desc_table_add(vq, desc, *num)) { >>>> virtqueue_kick(vq); >>>> - wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>>> - vb->balloon_page_chunk.chunk_num = 0; >>>> + if (busy_wait) >>>> + while (!virtqueue_get_buf(vq, &len) && >>>> + !virtqueue_is_broken(vq)) >>>> + cpu_relax(); >>>> + else >>>> + wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>> This is something I didn't previously notice. >>> As you always keep a single buffer in flight, you do not >>> really need indirect at all. Just add all descriptors >>> in the ring directly, then kick. >>> >>> E.g. >>> virtqueue_add_first >>> virtqueue_add_next >>> virtqueue_add_last >>> >>> ? >>> >>> You also want a flag to avoid allocations but there's no need to do it >>> per descriptor, set it on vq. >>> >> Without using the indirect table, I'm thinking about changing to use >> the standard sg (i.e. struct scatterlist), instead of vring_desc, so that >> we don't need to modify or add any new functions of virtqueue_add(). >> >> In this case, we will kmalloc an array of sgs in probe(), and we can add >> the sgs one by one to the vq, which won't trigger the allocation of an >> indirect table inside virtqueue_add(), and then kick when all are added. >> >> Best, >> Wei > And allocate headers too? This can work. API extensions aren't > necessarily a bad idea though. The API I suggest above is preferable > for the simple reason that it can work without INDIRECT flag > support in hypervisor. OK, probably we don't need to add a desc to the vq - we can just use the vq's desc, like this: int virtqueue_add_first(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, unsigned int *idx) { ... uint16_t desc_flags = in ? VRING_DESC_F_NEXT | VRING_DESC_F_WRITE : VRING_DESC_F_NEXT; vq->vring.desc[vq->free_head].addr = addr; vq->vring.desc[vq->free_head].len = len; vq->vring.desc[vq->free_head].flags = cpu_to_virtio16(_vq->vdev, flags); /* return to the caller the desc id */ *idx = vq->free_head; ... } int virtqueue_add_next(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, bool end, unsigned int *idx) { ... vq->vring.desc[*idx].next = vq->free_head; vq->vring.desc[vq->free_head].addr = addr; ... if (end) remove the VRING_DESC_F_NEXT flag } What do you think? We can also combine the two functions into one. Best, Wei -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752837AbdFVIiW (ORCPT ); Thu, 22 Jun 2017 04:38:22 -0400 Received: from mga11.intel.com ([192.55.52.93]:29311 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751874AbdFVIiU (ORCPT ); Thu, 22 Jun 2017 04:38:20 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,372,1493708400"; d="scan'208";a="117449298" Message-ID: <594B8287.6000706@intel.com> Date: Thu, 22 Jun 2017 16:40:39 +0800 From: Wei Wang User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "Michael S. Tsirkin" CC: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "david@redhat.com" , "Hansen, Dave" , "cornelia.huck@de.ibm.com" , "akpm@linux-foundation.org" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "liliang.opensource@gmail.com" , "riel@redhat.com" , "nilal@redhat.com" Subject: Re: [virtio-dev] Re: [PATCH v11 6/6] virtio-balloon: VIRTIO_BALLOON_F_CMD_VQ References: <1497004901-30593-1-git-send-email-wei.w.wang@intel.com> <1497004901-30593-7-git-send-email-wei.w.wang@intel.com> <20170620190343-mutt-send-email-mst@kernel.org> <5949E7C0.3050106@intel.com> <20170621151922-mutt-send-email-mst@kernel.org> In-Reply-To: <20170621151922-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 06/21/2017 08:28 PM, Michael S. Tsirkin wrote: > On Wed, Jun 21, 2017 at 11:28:00AM +0800, Wei Wang wrote: >> On 06/21/2017 12:18 AM, Michael S. Tsirkin wrote: >>> On Fri, Jun 09, 2017 at 06:41:41PM +0800, Wei Wang wrote: >>>> - if (!virtqueue_indirect_desc_table_add(vq, desc, num)) { >>>> + if (!virtqueue_indirect_desc_table_add(vq, desc, *num)) { >>>> virtqueue_kick(vq); >>>> - wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>>> - vb->balloon_page_chunk.chunk_num = 0; >>>> + if (busy_wait) >>>> + while (!virtqueue_get_buf(vq, &len) && >>>> + !virtqueue_is_broken(vq)) >>>> + cpu_relax(); >>>> + else >>>> + wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>> This is something I didn't previously notice. >>> As you always keep a single buffer in flight, you do not >>> really need indirect at all. Just add all descriptors >>> in the ring directly, then kick. >>> >>> E.g. >>> virtqueue_add_first >>> virtqueue_add_next >>> virtqueue_add_last >>> >>> ? >>> >>> You also want a flag to avoid allocations but there's no need to do it >>> per descriptor, set it on vq. >>> >> Without using the indirect table, I'm thinking about changing to use >> the standard sg (i.e. struct scatterlist), instead of vring_desc, so that >> we don't need to modify or add any new functions of virtqueue_add(). >> >> In this case, we will kmalloc an array of sgs in probe(), and we can add >> the sgs one by one to the vq, which won't trigger the allocation of an >> indirect table inside virtqueue_add(), and then kick when all are added. >> >> Best, >> Wei > And allocate headers too? This can work. API extensions aren't > necessarily a bad idea though. The API I suggest above is preferable > for the simple reason that it can work without INDIRECT flag > support in hypervisor. OK, probably we don't need to add a desc to the vq - we can just use the vq's desc, like this: int virtqueue_add_first(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, unsigned int *idx) { ... uint16_t desc_flags = in ? VRING_DESC_F_NEXT | VRING_DESC_F_WRITE : VRING_DESC_F_NEXT; vq->vring.desc[vq->free_head].addr = addr; vq->vring.desc[vq->free_head].len = len; vq->vring.desc[vq->free_head].flags = cpu_to_virtio16(_vq->vdev, flags); /* return to the caller the desc id */ *idx = vq->free_head; ... } int virtqueue_add_next(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, bool end, unsigned int *idx) { ... vq->vring.desc[*idx].next = vq->free_head; vq->vring.desc[vq->free_head].addr = addr; ... if (end) remove the VRING_DESC_F_NEXT flag } What do you think? We can also combine the two functions into one. Best, Wei From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dNxdD-0007vX-4a for qemu-devel@nongnu.org; Thu, 22 Jun 2017 04:38:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dNxdA-0004Pm-2c for qemu-devel@nongnu.org; Thu, 22 Jun 2017 04:38:27 -0400 Received: from mga05.intel.com ([192.55.52.43]:10465) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dNxd9-0004Nl-OF for qemu-devel@nongnu.org; Thu, 22 Jun 2017 04:38:23 -0400 Message-ID: <594B8287.6000706@intel.com> Date: Thu, 22 Jun 2017 16:40:39 +0800 From: Wei Wang MIME-Version: 1.0 References: <1497004901-30593-1-git-send-email-wei.w.wang@intel.com> <1497004901-30593-7-git-send-email-wei.w.wang@intel.com> <20170620190343-mutt-send-email-mst@kernel.org> <5949E7C0.3050106@intel.com> <20170621151922-mutt-send-email-mst@kernel.org> In-Reply-To: <20170621151922-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [virtio-dev] Re: [PATCH v11 6/6] virtio-balloon: VIRTIO_BALLOON_F_CMD_VQ List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "david@redhat.com" , "Hansen, Dave" , "cornelia.huck@de.ibm.com" , "akpm@linux-foundation.org" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "liliang.opensource@gmail.com" , "riel@redhat.com" , "nilal@redhat.com" On 06/21/2017 08:28 PM, Michael S. Tsirkin wrote: > On Wed, Jun 21, 2017 at 11:28:00AM +0800, Wei Wang wrote: >> On 06/21/2017 12:18 AM, Michael S. Tsirkin wrote: >>> On Fri, Jun 09, 2017 at 06:41:41PM +0800, Wei Wang wrote: >>>> - if (!virtqueue_indirect_desc_table_add(vq, desc, num)) { >>>> + if (!virtqueue_indirect_desc_table_add(vq, desc, *num)) { >>>> virtqueue_kick(vq); >>>> - wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>>> - vb->balloon_page_chunk.chunk_num = 0; >>>> + if (busy_wait) >>>> + while (!virtqueue_get_buf(vq, &len) && >>>> + !virtqueue_is_broken(vq)) >>>> + cpu_relax(); >>>> + else >>>> + wait_event(vb->acked, virtqueue_get_buf(vq, &len)); >>> This is something I didn't previously notice. >>> As you always keep a single buffer in flight, you do not >>> really need indirect at all. Just add all descriptors >>> in the ring directly, then kick. >>> >>> E.g. >>> virtqueue_add_first >>> virtqueue_add_next >>> virtqueue_add_last >>> >>> ? >>> >>> You also want a flag to avoid allocations but there's no need to do it >>> per descriptor, set it on vq. >>> >> Without using the indirect table, I'm thinking about changing to use >> the standard sg (i.e. struct scatterlist), instead of vring_desc, so that >> we don't need to modify or add any new functions of virtqueue_add(). >> >> In this case, we will kmalloc an array of sgs in probe(), and we can add >> the sgs one by one to the vq, which won't trigger the allocation of an >> indirect table inside virtqueue_add(), and then kick when all are added. >> >> Best, >> Wei > And allocate headers too? This can work. API extensions aren't > necessarily a bad idea though. The API I suggest above is preferable > for the simple reason that it can work without INDIRECT flag > support in hypervisor. OK, probably we don't need to add a desc to the vq - we can just use the vq's desc, like this: int virtqueue_add_first(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, unsigned int *idx) { ... uint16_t desc_flags = in ? VRING_DESC_F_NEXT | VRING_DESC_F_WRITE : VRING_DESC_F_NEXT; vq->vring.desc[vq->free_head].addr = addr; vq->vring.desc[vq->free_head].len = len; vq->vring.desc[vq->free_head].flags = cpu_to_virtio16(_vq->vdev, flags); /* return to the caller the desc id */ *idx = vq->free_head; ... } int virtqueue_add_next(struct virtqueue *_vq, uint64_t addr, uint32_t len, bool in, bool end, unsigned int *idx) { ... vq->vring.desc[*idx].next = vq->free_head; vq->vring.desc[vq->free_head].addr = addr; ... if (end) remove the VRING_DESC_F_NEXT flag } What do you think? We can also combine the two functions into one. Best, Wei