From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6477DC43218 for ; Thu, 25 Apr 2019 17:44:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E68AE20891 for ; Thu, 25 Apr 2019 17:44:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f//beVCT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729831AbfDYRob (ORCPT ); Thu, 25 Apr 2019 13:44:31 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:38490 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729507AbfDYRoa (ORCPT ); Thu, 25 Apr 2019 13:44:30 -0400 Received: by mail-pg1-f194.google.com with SMTP id j26so205132pgl.5 for ; Thu, 25 Apr 2019 10:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=1YlIFLmoqPb9gMcKTaBi8JbJDWAsLsG7A2SbBaFyuHg=; b=f//beVCTUJEmaA+Gj+UCUmQGRXMCI0QrfmmWxueltPpX8wx26YNzbwAxhjhwoJ+uth 80uklQUF+/HNJuYYSHf4Fq/mTES+guP9NOIULYnhesnI8GlHWQCwZdvW7Fg3BHvnAvNI DEeFskrb0wKmDBvxh19/RHCLL2bZL9aosA6J/WoDoaewyC7sd014ylK9WGb1O+ZMmzYt g1Ul0gv4/+hM8W2SXDB0OcGDfn4ZR7WCmeX0guV/fbOa1gl+N2SAt3R0HKass6srzL8Z sEkGl8O9ysbqy4tHv049KsBm/8ed1LWBzxOSX4VUnf+TCqeJbmuHqY5Nz+MjTn/C7q+o tZZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=1YlIFLmoqPb9gMcKTaBi8JbJDWAsLsG7A2SbBaFyuHg=; b=klWwkc9LGqqP73XVf64auUd6rCPcxRCkQJ+pWEzQK8qkUd7xnEHDgS0zN+OxH1YdgT ivrC5uzkM0SZaEFJvbX6dK3D9C6yxXJDG3vC4weKdAshOj4y/NATzQyCkXW+yyMZcG5X d51YHfo/heE7BBWUYGKxDtXQX95b4ZETdP74kR7wwqDNU1pWuv9FBdFGdAhMuXDj+LRk dE1xKd/adpWwMv/yJbGOSvB2/CET8q+eaaos2RUFfUWiYQ2mYFZDVWzqXl2hkU2OCF30 n57rzbw+7CoTaTnvmujzwIsDdBEC/8Iv8/5AyGYYfbTFABH1gURGkQxxy30LWZvqkkiA 8PUA== X-Gm-Message-State: APjAAAXhsKKBaiFo0n8EwZuzIEAjtij1eOlVeWk8WRY7OJaVaWbkUVTy znWRkWKGZSWUqp1T2r6NsndPXbWb X-Google-Smtp-Source: APXvYqzK+d6PxOJE4xCZED2tUFmn4Xhl+dYIZr4R1bRtSkkcZ3r91LDkFdFRrQ4f5RWYmM0rQFbJAg== X-Received: by 2002:a65:6658:: with SMTP id z24mr9220085pgv.323.1556214269741; Thu, 25 Apr 2019 10:44:29 -0700 (PDT) Received: from ?IPv6:2601:282:800:fd80:7cc1:255c:d029:91f7? ([2601:282:800:fd80:7cc1:255c:d029:91f7]) by smtp.googlemail.com with ESMTPSA id c18sm52013614pfc.0.2019.04.25.10.44.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Apr 2019 10:44:28 -0700 (PDT) Subject: Re: virtio_net: suspicious RCU usage with xdp To: Jesper Dangaard Brouer , "Michael S. Tsirkin" Cc: Toshiaki Makita , =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , "netdev@vger.kernel.org" , Jason Wang References: <20190424132533-mutt-send-email-mst@kernel.org> <20190425130319-mutt-send-email-mst@kernel.org> <20190425194117.66093851@carbon> From: David Ahern Message-ID: <8a0390d8-9d08-7ac9-6cc7-6d0612062226@gmail.com> Date: Thu, 25 Apr 2019 11:44:27 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190425194117.66093851@carbon> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 4/25/19 11:41 AM, Jesper Dangaard Brouer wrote: > On Thu, 25 Apr 2019 13:03:39 -0400 > "Michael S. Tsirkin" wrote: > >> On Thu, Apr 25, 2019 at 01:58:48PM +0900, Toshiaki Makita wrote: >>> On 2019/04/25 2:37, Michael S. Tsirkin wrote: >>>> On Wed, Apr 24, 2019 at 11:13:42AM -0600, David Ahern wrote: >>>>> seeing an RCU warning testing xdp with virtio net. net-next as of commit >>>>> b2f97f7de2f6a4df8e431330cf467576486651c5. No obvious changes so hoping >>>>> this rings a bell with someone else. >>>>> >>>>> >>>>> [ 121.990304] ============================= >>>>> [ 121.991488] WARNING: suspicious RCU usage >>>>> [ 121.992392] 5.1.0-rc5+ #60 Not tainted >>>>> [ 121.993220] ----------------------------- >>>>> [ 121.994158] /home/dsa/kernel-3.git/drivers/net/virtio_net.c:516 >>>>> suspicious rcu_dereference_check() usage! >>>>> [ 121.996284] >>>>> other info that might help us debug this: >>>>> >>>>> [ 121.997988] >>>>> rcu_scheduler_active = 2, debug_locks = 1 >>>>> [ 121.999321] no locks held by swapper/1/0. >>>>> [ 122.000328] >>>>> stack backtrace: >>>>> [ 122.001253] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.1.0-rc5+ #60 >>>>> [ 122.002474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>>> BIOS 1.11.1-1 04/01/2014 >>>>> [ 122.004141] Call Trace: >>>>> [ 122.004651] >>>>> [ 122.005082] dump_stack+0x7e/0xbb >>>>> [ 122.005757] lockdep_rcu_suspicious+0x102/0x10b >>>>> [ 122.006654] virtnet_xdp_xmit+0x104/0x4fe >>>>> [ 122.007447] ? kasan_check_read+0x11/0x13 >>>>> [ 122.008267] ? mergeable_rx_buffer_size_show+0x163/0x163 >>>>> [ 122.009299] ? __asan_loadN+0xf/0x11 >>>>> [ 122.010010] ? pvclock_clocksource_read+0xfa/0x189 >>>>> [ 122.010975] bq_xmit_all+0xdc/0x358 >>>>> [ 122.011699] __dev_map_flush+0xc2/0xef >>>>> [ 122.012472] xdp_do_flush_map+0x5b/0x74 >>>>> [ 122.013238] virtnet_poll+0x58f/0x679 >>>> >>>> Well virtnet_xdp_xmit seems to be called from .ndo_xdp_xmit >>>> and that isn't in an RCU read-side critical section. >>>> >>>> Looks like we just need to add RCU read lock/unlock. >>>> Like the below perhaps? >>>> >>>> This issue was introduced by 8dcc5b0ab0 however I find it >>> >>> Probably not 8dcc5b0ab0, but 5d053f9da431 ("bpf: devmap prepare xdp >>> frames for bulking"). >>> >>>> inelegant that we need to do checks in each driver, >>>> and add RCU locks just for a startup initialization issue. >>>> Can't XDP core make sure the callback isn't invoked >>>> at an inappropriate time instead? >>> >>> Before commit 5d053f9da431, .ndo_xdp_xmit() should have always been >>> called under RCU. After the commit, xdp_do_flush_map() also can trigger >>> .ndo_xdp_xmit() but we forgot to add RCU read lock there? >>> I guess veth has the same problem and I feel like it should be fixed in >>> __dev_map_flush(). dev_map_flush_old() needs to be cared too. >> >> I don't have a problem either way. Jesper, what do you think? > > It does sound like my commit 5d053f9da431 ("bpf: devmap prepare xdp > frames for bulking") introduced this issue. I guess we can add the RCU > section to xdp_do_flush_map(), and then also verify that the devmap > (and cpumap) take-down code also have appropriate RCU sections (which > they should have). > > Another requirement for calling .ndo_xdp_xmit is running under NAPI > protection, is that still satisifed for veth? > (even when invoked via xdp_do_flush_map()). > virtio_net hits this because of: xdp_prog = rcu_dereference(rq->xdp_prog); in its ndo_xdp_xmit. Scanning ndo_xdp_xmit for other nics does not show this same check. Is it really required? If so, why don't other drivers do it?