From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE7CCC54E60 for ; Tue, 19 Mar 2024 04:59:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gb3w8EwcQ7DboNmsc6M4OJG+iPTp55kiUgq8Uz7Pcok=; b=zYBhv39VSM0YSA BfgOWp/vBVoyP803SXhIqXjdkiyg6KrMR1SHunNfn7FaX1Bca7K37jiPdY+YyubqkXKBLjNgIE6/a YbeUWoah1nzAWi1rn42DSjIJW0LM4xRzn7lTnr0UtTUdEScaNMW8nHLKJBWzoNMU0WPveGz5pPq6b 8vt1gxA6vPhWeGzOQ26+Vo5zW8sVf3uZ1LzktRXKwk/FpbW/a9agq6xp/N2MYeReGgF9ItsugxJat wd0xyTRwJ+PjrcJz0u8VEAoecmb1eiShivXGhOD/fMeFPKg7dUm9O8eB1HLNSaiWY1xy7uZpmnWcd jxJszbCKmFgRuFAYxCEw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmRZc-0000000BHMn-2zBO; Tue, 19 Mar 2024 04:59:40 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmRZZ-0000000BHM3-2DOO for linux-arm-kernel@lists.infradead.org; Tue, 19 Mar 2024 04:59:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710824374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Eu1fZTuW5w55lZyjKAedr2/F7yOPJnQYv8yLHqlcanI=; b=BBDvvwdLmXqDJj+2E39uwKMLeihITmKHqJ9mGzt+9HG3pKeoXSmZiyBipfY9qy5a6vX/AG 6I/+iDmnN/+seNReqXmcA1hLNzZ4o8M1G0H9UhN76/Hf6oG4E9QZn28VbMb8kaUW6QMc68 5jVKgkTLUPKcVxLA1MuMdRemMMt2DHg= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-325-C556n_yhMtObPXSzyqyQew-1; Tue, 19 Mar 2024 00:59:31 -0400 X-MC-Unique: C556n_yhMtObPXSzyqyQew-1 Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-6e7149bc12fso1641859b3a.3 for ; Mon, 18 Mar 2024 21:59:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710824370; x=1711429170; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Eu1fZTuW5w55lZyjKAedr2/F7yOPJnQYv8yLHqlcanI=; b=mXqgYU5ExV38U7z6SHhpyw95ZAmQCcGjGUuNtoIGJ4IL4lFMekd0mS2/RsyEFurv2C uW9J2jiTR6ghmPG1dBCLk5ABsFDocwUM01HiZZCTWACQhNrSr1FYZD8aP7erD37O69PX 4xElUHLVqDHewEOapFdwDtQac0aSDPJBvJuf4cLuIpUkbnsSnhyWe1jTY7YzAuBF5/ev p4CMTdGOkJKnvbEsFHdQatPHtGQvKrnDRHCb3RTQSYEuuNv7XmXpBEliM6ycmLyWp52M t/ZZJ63vByNdwfZO8ZnJjxy0EH/9N01OyKuYof1uEuMAbK8Orx1SI9ny00ZKfLnvR9kT BTYw== X-Forwarded-Encrypted: i=1; AJvYcCWTGndPjgowToJ40f4xaIUCaOVkG1WQMcoemDfWximOhKOdfd0nwRxITw+QJYeeY4Sjr6/BQbB35538gYfa+lL2BA4XXZl4oJYa0XmHSSj12QkhuwI= X-Gm-Message-State: AOJu0YzLiBfy3Gv5eILtM0l5OCVGdLj8iSZmJFJ3bpRDzfxHCCJXKmHA 4zTEx5t29l4rTOf1SmpYwcrejgJ5Mn2TSzFxYzbdSmXdB20nA82ZJnPXWI/ZK9OlTNAeJlXDT8U 3A6znHX+zF70BFBeF23CwuNMF9D/bpXVVXOoabXhP3d77m7Ve/Bnmz41frkwjRkEseIpCfYh1 X-Received: by 2002:aa7:85c4:0:b0:6e7:2199:6791 with SMTP id z4-20020aa785c4000000b006e721996791mr5561433pfn.21.1710824370498; Mon, 18 Mar 2024 21:59:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDQe1IN2wAF4VXFN5LqilMUu9jTHpCmZYTCvncFnnI71NO8xd5+5kCQNJYibUaJASRkbY8dg== X-Received: by 2002:aa7:85c4:0:b0:6e7:2199:6791 with SMTP id z4-20020aa785c4000000b006e721996791mr5561425pfn.21.1710824370136; Mon, 18 Mar 2024 21:59:30 -0700 (PDT) Received: from [192.168.68.51] ([43.252.115.31]) by smtp.gmail.com with ESMTPSA id t23-20020a63b257000000b005dcbb699abfsm8145276pgo.34.2024.03.18.21.59.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Mar 2024 21:59:29 -0700 (PDT) Message-ID: <35a6bcef-27cf-4626-a41d-9ec0a338fe28@redhat.com> Date: Tue, 19 Mar 2024 14:59:23 +1000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] virtio_ring: Fix the stale index in available ring To: Will Deacon Cc: virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, yihyu@redhat.com, shan.gavin@gmail.com, linux-arm-kernel@lists.infradead.org, Catalin Marinas , mochs@nvidia.com References: <20240314074923.426688-1-gshan@redhat.com> <20240318165924.GA1824@willie-the-truck> From: Gavin Shan In-Reply-To: <20240318165924.GA1824@willie-the-truck> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240318_215937_694722_D9C18692 X-CRM114-Status: GOOD ( 29.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 3/19/24 02:59, Will Deacon wrote: > On Thu, Mar 14, 2024 at 05:49:23PM +1000, Gavin Shan wrote: >> The issue is reported by Yihuang Yu who have 'netperf' test on >> NVidia's grace-grace and grace-hopper machines. The 'netperf' >> client is started in the VM hosted by grace-hopper machine, >> while the 'netperf' server is running on grace-grace machine. >> >> The VM is started with virtio-net and vhost has been enabled. >> We observe a error message spew from VM and then soft-lockup >> report. The error message indicates the data associated with >> the descriptor (index: 135) has been released, and the queue >> is marked as broken. It eventually leads to the endless effort >> to fetch free buffer (skb) in drivers/net/virtio_net.c::start_xmit() >> and soft-lockup. The stale index 135 is fetched from the available >> ring and published to the used ring by vhost, meaning we have >> disordred write to the available ring element and available index. >> >> /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ >> -accel kvm -machine virt,gic-version=host \ >> : \ >> -netdev tap,id=vnet0,vhost=on \ >> -device virtio-net-pci,bus=pcie.8,netdev=vnet0,mac=52:54:00:f1:26:b0 \ >> >> [ 19.993158] virtio_net virtio1: output.0:id 135 is not a head! >> >> Fix the issue by replacing virtio_wmb(vq->weak_barriers) with stronger >> virtio_mb(false), equivalent to replaced 'dmb' by 'dsb' instruction on >> ARM64. It should work for other architectures, but performance loss is >> expected. >> >> Cc: stable@vger.kernel.org >> Reported-by: Yihuang Yu >> Signed-off-by: Gavin Shan >> --- >> drivers/virtio/virtio_ring.c | 12 +++++++++--- >> 1 file changed, 9 insertions(+), 3 deletions(-) >> >> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c >> index 49299b1f9ec7..7d852811c912 100644 >> --- a/drivers/virtio/virtio_ring.c >> +++ b/drivers/virtio/virtio_ring.c >> @@ -687,9 +687,15 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, >> avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); >> vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); >> >> - /* Descriptors and available array need to be set before we expose the >> - * new available array entries. */ >> - virtio_wmb(vq->weak_barriers); >> + /* >> + * Descriptors and available array need to be set before we expose >> + * the new available array entries. virtio_wmb() should be enough >> + * to ensuere the order theoretically. However, a stronger barrier >> + * is needed by ARM64. Otherwise, the stale data can be observed >> + * by the host (vhost). A stronger barrier should work for other >> + * architectures, but performance loss is expected. >> + */ >> + virtio_mb(false); >> vq->split.avail_idx_shadow++; >> vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, >> vq->split.avail_idx_shadow); > > Replacing a DMB with a DSB is _very_ unlikely to be the correct solution > here, especially when ordering accesses to coherent memory. > > In practice, either the larger timing different from the DSB or the fact > that you're going from a Store->Store barrier to a full barrier is what > makes things "work" for you. Have you tried, for example, a DMB SY > (e.g. via __smb_mb()). > > We definitely shouldn't take changes like this without a proper > explanation of what is going on. > Thanks for your comments, Will. Yes, DMB should work for us. However, it seems this instruction has issues on NVidia's grace-hopper. It's hard for me to understand how DMB and DSB works from hardware level. I agree it's not the solution to replace DMB with DSB before we fully understand the root cause. I tried the possible replacement like below. __smp_mb() can avoid the issue like __mb() does. __ndelay(10) can avoid the issue, but __ndelay(9) doesn't. static inline int virtqueue_add_split(struct virtqueue *_vq, ...) { : /* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); /* Descriptors and available array need to be set before we expose the * new available array entries. */ // Broken: virtio_wmb(vq->weak_barriers); // Broken: __dma_mb(); // Work: __mb(); // Work: __smp_mb(); // Work: __ndelay(100); // Work: __ndelay(10); // Broken: __ndelay(9); vq->split.avail_idx_shadow++; vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->split.avail_idx_shadow); vq->num_added++; pr_debug("Added buffer head %i to %p\n", head, vq); END_USE(vq); : } I also tried to measure the consumed time for various barrier-relative instructions using ktime_get_ns() which should have consumed most of the time. __smb_mb() is slower than __smp_wmb() but faster than __mb() Instruction Range of used time in ns ---------------------------------------------- __smp_wmb() [32 1128032] __smp_mb() [32 1160096] __mb() [32 1162496] Thanks, Gavin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel