From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E581DC54E60 for ; Tue, 19 Mar 2024 08:08:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=drqF1xhFjPZK6ltklj+Aok1xc/q8PiENFXJ03dTAeW0=; b=g/LazSkk33TmfH uLOSDqeOzxMsCD/zvuG+v2d1dFImj8MUjSDiThT5HDyfPoWL/bHRxTIEL3JtIxRC4x/7xTPsdXq4S 45ksLOhfMrKypJTqJ1KuE8KLrCESJoAW7FaPqw+/12T2FVPrZ0BYeFizVSkTA3etZXfWpOaWRxK1A O4uzGkYbPOZ9qJ2XagjwD5Iz/JQxNstFysIOVJLcOLVYmA4aIXMKxyS1UfxHjzxM4j1MrnqjlaqA2 zeqV+a3dPKhYUP33cx8NqWvg1FpyFzepCBXiLVY708t6YyHZQkellNH6KlJSZ6aszX9d5MnnqQBOY A2B0QTim8A1WJ0hAYjvA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmUWW-0000000BoGb-13Wc; Tue, 19 Mar 2024 08:08:40 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmUWT-0000000BoEz-0ypI for linux-arm-kernel@lists.infradead.org; Tue, 19 Mar 2024 08:08:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710835716; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ju8gWBaWN+2UIqDfiEjPtIBulqNS5l/Hxkm05DauZf8=; b=UNsuN4PHvvJL2k+EgBBKJRf236HTFGHc2qpDJKlNsQEuZ7kUswgR0IfAlFEYw4BaFeTeef KiofATAjTdzg8LY3knlFuQxJPRU74M4nQE9woOsmg08xhyHy6C58lOmVU/iHC2SZrlr9O9 zCefs5kVyhgMezhlRI/hkNs/X4Iohbg= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-605-VY6tW_oCM0qDo3Vy4ugBTQ-1; Tue, 19 Mar 2024 04:08:35 -0400 X-MC-Unique: VY6tW_oCM0qDo3Vy4ugBTQ-1 Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-6e6bab4b84dso5109800b3a.0 for ; Tue, 19 Mar 2024 01:08:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710835714; x=1711440514; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ju8gWBaWN+2UIqDfiEjPtIBulqNS5l/Hxkm05DauZf8=; b=Sfc83ljlE3w0gBM8PCjSod8cROrh9HPOiQl+9y/EK37jZj7JeG5qLBH8f8Y+hOzYf4 I1eZ08bM3KnqX5nXkXS9JWbQp9wdLNfekKF7XVxVIkQ5cMSU0Lo/W9vupZCSe681gI7T rNG+EPPOTkQTG1zYe46fyR+vuIMyMi8L4x4JLX8VriitTlpBaO3oh5Q3K/wKC1W+dbv6 0FGJENISyAGMqgabpmUVdO2432IEKY2t+DbFevkTKsyUMN2vJpvC4K8CZTynaU2bPCJu 4u6d8GJgXZZFi10gxtpDSG+1T+/kB1fBdnKcspFvsE83G1F5FS+NUKwUfcXmRDikzgMl HJXg== X-Forwarded-Encrypted: i=1; AJvYcCVQtCo0v+5oU6Zn1GISnBtOaMN20t+JtpY5geNrcKvgBNSnpBoGLwwbDZ4r7/ya4qXbdqoF0sPmoDCJ0/dwM8f6NmtAh/A6aFWQPFr3ZWgqySSym0M= X-Gm-Message-State: AOJu0Yz7IAzVNimP0o1iScVXpf0/7L3kgQa4EEl8F+vv1xTYsxptMdR2 TpssOnOQN+gHkGlGwOOGK6ZxJ43GsDhttXcoI3BdTzN4uPpZBNLdNu6iyDSYA1S6jvlBCwmTYGi N6Bq4i86lCldF/IYahcMel+20N1+DAoOxINbNGXWd87o9LsNQoIYDQLagjK2joPL7E9nNSdnj X-Received: by 2002:a05:6a20:3955:b0:1a3:6704:8885 with SMTP id r21-20020a056a20395500b001a367048885mr1371130pzg.51.1710835713761; Tue, 19 Mar 2024 01:08:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFk7L/5R++tIec7RGJ5dVuOOSuByzVSgvORsavBQdMwSBFir5LXMrz07yEfky6wE6VMoQutNA== X-Received: by 2002:a05:6a20:3955:b0:1a3:6704:8885 with SMTP id r21-20020a056a20395500b001a367048885mr1371095pzg.51.1710835712980; Tue, 19 Mar 2024 01:08:32 -0700 (PDT) Received: from [192.168.68.51] ([43.252.115.31]) by smtp.gmail.com with ESMTPSA id h14-20020a170902680e00b001dda608484asm10932340plk.57.2024.03.19.01.08.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 19 Mar 2024 01:08:32 -0700 (PDT) Message-ID: Date: Tue, 19 Mar 2024 18:08:27 +1000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] virtio_ring: Fix the stale index in available ring To: "Michael S. Tsirkin" Cc: Will Deacon , virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, yihyu@redhat.com, shan.gavin@gmail.com, linux-arm-kernel@lists.infradead.org, Catalin Marinas , mochs@nvidia.com References: <20240314074923.426688-1-gshan@redhat.com> <20240318165924.GA1824@willie-the-truck> <35a6bcef-27cf-4626-a41d-9ec0a338fe28@redhat.com> <20240319020905-mutt-send-email-mst@kernel.org> <9b3030d1-cb2c-4ce0-8b24-1074b616fc84@redhat.com> <20240319024025-mutt-send-email-mst@kernel.org> <20240319030505-mutt-send-email-mst@kernel.org> From: Gavin Shan In-Reply-To: <20240319030505-mutt-send-email-mst@kernel.org> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240319_010837_403026_DC92E5E3 X-CRM114-Status: GOOD ( 27.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 3/19/24 17:09, Michael S. Tsirkin wrote: > On Tue, Mar 19, 2024 at 04:49:50PM +1000, Gavin Shan wrote: >> >> On 3/19/24 16:43, Michael S. Tsirkin wrote: >>> On Tue, Mar 19, 2024 at 04:38:49PM +1000, Gavin Shan wrote: >>>> On 3/19/24 16:09, Michael S. Tsirkin wrote: >>>> >>>>>>>> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c >>>>>>>> index 49299b1f9ec7..7d852811c912 100644 >>>>>>>> --- a/drivers/virtio/virtio_ring.c >>>>>>>> +++ b/drivers/virtio/virtio_ring.c >>>>>>>> @@ -687,9 +687,15 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, >>>>>>>> avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); >>>>>>>> vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); >>>>>>>> - /* Descriptors and available array need to be set before we expose the >>>>>>>> - * new available array entries. */ >>>>>>>> - virtio_wmb(vq->weak_barriers); >>>>>>>> + /* >>>>>>>> + * Descriptors and available array need to be set before we expose >>>>>>>> + * the new available array entries. virtio_wmb() should be enough >>>>>>>> + * to ensuere the order theoretically. However, a stronger barrier >>>>>>>> + * is needed by ARM64. Otherwise, the stale data can be observed >>>>>>>> + * by the host (vhost). A stronger barrier should work for other >>>>>>>> + * architectures, but performance loss is expected. >>>>>>>> + */ >>>>>>>> + virtio_mb(false); >>>>>>>> vq->split.avail_idx_shadow++; >>>>>>>> vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, >>>>>>>> vq->split.avail_idx_shadow); >>>>>>> >>>>>>> Replacing a DMB with a DSB is _very_ unlikely to be the correct solution >>>>>>> here, especially when ordering accesses to coherent memory. >>>>>>> >>>>>>> In practice, either the larger timing different from the DSB or the fact >>>>>>> that you're going from a Store->Store barrier to a full barrier is what >>>>>>> makes things "work" for you. Have you tried, for example, a DMB SY >>>>>>> (e.g. via __smb_mb()). >>>>>>> >>>>>>> We definitely shouldn't take changes like this without a proper >>>>>>> explanation of what is going on. >>>>>>> >>>>>> >>>>>> Thanks for your comments, Will. >>>>>> >>>>>> Yes, DMB should work for us. However, it seems this instruction has issues on >>>>>> NVidia's grace-hopper. It's hard for me to understand how DMB and DSB works >>>>>> from hardware level. I agree it's not the solution to replace DMB with DSB >>>>>> before we fully understand the root cause. >>>>>> >>>>>> I tried the possible replacement like below. __smp_mb() can avoid the issue like >>>>>> __mb() does. __ndelay(10) can avoid the issue, but __ndelay(9) doesn't. >>>>>> >>>>>> static inline int virtqueue_add_split(struct virtqueue *_vq, ...) >>>>>> { >>>>>> : >>>>>> /* Put entry in available array (but don't update avail->idx until they >>>>>> * do sync). */ >>>>>> avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); >>>>>> vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); >>>>>> >>>>>> /* Descriptors and available array need to be set before we expose the >>>>>> * new available array entries. */ >>>>>> // Broken: virtio_wmb(vq->weak_barriers); >>>>>> // Broken: __dma_mb(); >>>>>> // Work: __mb(); >>>>>> // Work: __smp_mb(); >>>>>> // Work: __ndelay(100); >>>>>> // Work: __ndelay(10); >>>>>> // Broken: __ndelay(9); >>>>>> >>>>>> vq->split.avail_idx_shadow++; >>>>>> vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, >>>>>> vq->split.avail_idx_shadow); >>>>> >>>>> What if you stick __ndelay here? >>>>> >>>> >>>> /* Put entry in available array (but don't update avail->idx until they >>>> * do sync). */ >>>> avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); >>>> vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); >>>> >>>> /* Descriptors and available array need to be set before we expose the >>>> * new available array entries. */ >>>> virtio_wmb(vq->weak_barriers); >>>> vq->split.avail_idx_shadow++; >>>> vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, >>>> vq->split.avail_idx_shadow); >>>> /* Try __ndelay(x) here as Michael suggested >>>> * >>>> * Work: __ndelay(200); possiblly make it hard to reproduce >>>> * Broken: __ndelay(100); >>>> * Broken: __ndelay(20); >>>> * Broken: __ndelay(10); >>>> */ >>>> __ndelay(200); >>> >>> So we see that just changing the timing masks the race. >>> What are you using on the host side? vhost or qemu? >>> >> >> __ndelay(200) may make the issue harder to be reproduce as I understand. >> More delays here will give vhost relief, reducing the race. >> >> The issue is only reproducible when vhost is turned on. Otherwise, we >> aren't able to hit the issue. >> >> -netdev tap,id=vnet0,vhost=true,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \ >> -device virtio-net-pci,bus=pcie.8,netdev=vnet0,mac=52:54:00:f1:26:b0 > > > Given it's vhost, it's also possible that the issue is host side. > I wonder what happens if we stick a delay or a stronger barrier > in vhost.c - either here: > > /* Make sure buffer is written before we update index. */ > smp_wmb(); > > > or here: > > /* Only get avail ring entries after they have been > * exposed by guest. > */ > smp_rmb(); > > ? > It's possible. However, I still can hit the issue after both of them are replaed with '__mb()'. So the issue seems on the guest side, where the written data isn't observed in time by the CPU on far end (for vhost worker). diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 045f666b4f12..327b68d21b02 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2529,7 +2529,8 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, /* Only get avail ring entries after they have been * exposed by guest. */ - smp_rmb(); + // smp_rmb(); + __mb(); } /* Grab the next descriptor number they're advertising, and increment @@ -2703,7 +2704,9 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, r = __vhost_add_used_n(vq, heads, count); /* Make sure buffer is written before we update index. */ - smp_wmb(); + // smp_wmb(); + __mb(); + From the guest: [ 14.102608] virtio_net virtio0: output.0:id 80 is not a head! >>>> >>>>> >>>>>> vq->num_added++; >>>>>> >>>>>> pr_debug("Added buffer head %i to %p\n", head, vq); >>>>>> END_USE(vq); >>>>>> : >>>>>> } >>>>>> >>>>>> I also tried to measure the consumed time for various barrier-relative instructions using >>>>>> ktime_get_ns() which should have consumed most of the time. __smb_mb() is slower than >>>>>> __smp_wmb() but faster than __mb() >>>>>> >>>>>> Instruction Range of used time in ns >>>>>> ---------------------------------------------- >>>>>> __smp_wmb() [32 1128032] >>>>>> __smp_mb() [32 1160096] >>>>>> __mb() [32 1162496] >>>>>> Thanks, Gavin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel