From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D8AC43219 for ; Thu, 25 Apr 2019 18:54:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 68F932077C for ; Thu, 25 Apr 2019 18:54:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mr2xusPM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726638AbfDYSyn (ORCPT ); Thu, 25 Apr 2019 14:54:43 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:45267 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725900AbfDYSyn (ORCPT ); Thu, 25 Apr 2019 14:54:43 -0400 Received: by mail-pg1-f194.google.com with SMTP id y3so276808pgk.12 for ; Thu, 25 Apr 2019 11:54:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L49s+kTrcEkEgFjBCkBs3m9H5LKRxpW2+N5HCmlOFHU=; b=mr2xusPMDR67Dt2qYTkVUZdG6/KbOslVBH1sJGfFoVcXlMHSI7b6kJ5hF3notDv23N /HBF/HaoKu8lf24RyTF0/ZWdAcQxWnbDlnt+ZhKTJ2Wlmcu3j9qz7ol92l1gXSESF39M xmplg4OolNg76LIxCQDERnXcFsxlxkrn0vaAQB7+656yaD2MCWWhNhVlHvTEQCXn8CQD l/Ss2Fht0NA3bfiPfUFqBZqFrvzXMtph1nH1cRSaf+Vb3qJwEDuguKdYYb3tiPqCW62a /D3y/J4V15r41x57a5Pkf45KkJIxuGGAYJcMzCKMpankdIIWtUw+Rw6QN4IbDr3XbWW+ U9zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L49s+kTrcEkEgFjBCkBs3m9H5LKRxpW2+N5HCmlOFHU=; b=CnnHiizLOzO5oEp7sp8Gb3MO5Wxe+sWbMfSn+sMuPj52XufDshH1q+q7uT0ZBOe+jg MKbNdkdgzOJHk8bOL7wrfcRtgHB8h4X2adbhL0tjZ76xk7iPHq3/bzL7GcsrpQztZGZc vk1RrBWfZ0KStINYvBI7NWtS1KFa2IWjbZ1tDDueTqYAYuIt4tOnI+Elxz7yj5eonYd8 1d8FZcmAlczhHZDIVwmPf+NHynZi73JHh8zQr0bV3lFLbkRQOe470IsB4OzHtHoVdMaa FJOJAKEpoiYI/KOqhzgXWvd7ppm4n14PMH4JsLGGs+GWVKwJjo4tDuHOETG6nHHdsZrB QvNg== X-Gm-Message-State: APjAAAW3HepuXdDuleqYoZLi7ADmFVptV2pP7SKx2ohInsdfOj/VmNbG TNX7N72X2Yw4yk2N5AZ4wCc= X-Google-Smtp-Source: APXvYqzr6pqCJC08q3VLJ/UgNxfsd6fQIOuOGrERdqlQIXBrGycsKfnga1JDNSBYlW1DEIv1Ec8sIw== X-Received: by 2002:a65:5089:: with SMTP id r9mr38672495pgp.14.1556218482504; Thu, 25 Apr 2019 11:54:42 -0700 (PDT) Received: from localhost ([192.55.54.44]) by smtp.gmail.com with ESMTPSA id a129sm37008267pfa.152.2019.04.25.11.54.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Apr 2019 11:54:42 -0700 (PDT) Date: Thu, 25 Apr 2019 20:54:11 +0200 From: Maciej Fijalkowski To: David Ahern Cc: Jesper Dangaard Brouer , "Michael S. Tsirkin" , Toshiaki Makita , Toke =?ISO-8859-1?Q?H=F8iland-J=F8rgensen?= , "netdev@vger.kernel.org" , Jason Wang Subject: Re: virtio_net: suspicious RCU usage with xdp Message-ID: <20190425205349.0000137c@gmail.com> In-Reply-To: <8a0390d8-9d08-7ac9-6cc7-6d0612062226@gmail.com> References: <20190424132533-mutt-send-email-mst@kernel.org> <20190425130319-mutt-send-email-mst@kernel.org> <20190425194117.66093851@carbon> <8a0390d8-9d08-7ac9-6cc7-6d0612062226@gmail.com> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu, 25 Apr 2019 11:44:27 -0600 David Ahern wrote: > On 4/25/19 11:41 AM, Jesper Dangaard Brouer wrote: > > On Thu, 25 Apr 2019 13:03:39 -0400 > > "Michael S. Tsirkin" wrote: > > > >> On Thu, Apr 25, 2019 at 01:58:48PM +0900, Toshiaki Makita wrote: > >>> On 2019/04/25 2:37, Michael S. Tsirkin wrote: > >>>> On Wed, Apr 24, 2019 at 11:13:42AM -0600, David Ahern wrote: > >>>>> seeing an RCU warning testing xdp with virtio net. net-next as of commit > >>>>> b2f97f7de2f6a4df8e431330cf467576486651c5. No obvious changes so hoping > >>>>> this rings a bell with someone else. > >>>>> > >>>>> > >>>>> [ 121.990304] ============================= > >>>>> [ 121.991488] WARNING: suspicious RCU usage > >>>>> [ 121.992392] 5.1.0-rc5+ #60 Not tainted > >>>>> [ 121.993220] ----------------------------- > >>>>> [ 121.994158] /home/dsa/kernel-3.git/drivers/net/virtio_net.c:516 > >>>>> suspicious rcu_dereference_check() usage! > >>>>> [ 121.996284] > >>>>> other info that might help us debug this: > >>>>> > >>>>> [ 121.997988] > >>>>> rcu_scheduler_active = 2, debug_locks = 1 > >>>>> [ 121.999321] no locks held by swapper/1/0. > >>>>> [ 122.000328] > >>>>> stack backtrace: > >>>>> [ 122.001253] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.1.0-rc5+ #60 > >>>>> [ 122.002474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > >>>>> BIOS 1.11.1-1 04/01/2014 > >>>>> [ 122.004141] Call Trace: > >>>>> [ 122.004651] > >>>>> [ 122.005082] dump_stack+0x7e/0xbb > >>>>> [ 122.005757] lockdep_rcu_suspicious+0x102/0x10b > >>>>> [ 122.006654] virtnet_xdp_xmit+0x104/0x4fe > >>>>> [ 122.007447] ? kasan_check_read+0x11/0x13 > >>>>> [ 122.008267] ? mergeable_rx_buffer_size_show+0x163/0x163 > >>>>> [ 122.009299] ? __asan_loadN+0xf/0x11 > >>>>> [ 122.010010] ? pvclock_clocksource_read+0xfa/0x189 > >>>>> [ 122.010975] bq_xmit_all+0xdc/0x358 > >>>>> [ 122.011699] __dev_map_flush+0xc2/0xef > >>>>> [ 122.012472] xdp_do_flush_map+0x5b/0x74 > >>>>> [ 122.013238] virtnet_poll+0x58f/0x679 > >>>> > >>>> Well virtnet_xdp_xmit seems to be called from .ndo_xdp_xmit > >>>> and that isn't in an RCU read-side critical section. > >>>> > >>>> Looks like we just need to add RCU read lock/unlock. > >>>> Like the below perhaps? > >>>> > >>>> This issue was introduced by 8dcc5b0ab0 however I find it > >>> > >>> Probably not 8dcc5b0ab0, but 5d053f9da431 ("bpf: devmap prepare xdp > >>> frames for bulking"). > >>> > >>>> inelegant that we need to do checks in each driver, > >>>> and add RCU locks just for a startup initialization issue. > >>>> Can't XDP core make sure the callback isn't invoked > >>>> at an inappropriate time instead? > >>> > >>> Before commit 5d053f9da431, .ndo_xdp_xmit() should have always been > >>> called under RCU. After the commit, xdp_do_flush_map() also can trigger > >>> .ndo_xdp_xmit() but we forgot to add RCU read lock there? > >>> I guess veth has the same problem and I feel like it should be fixed in > >>> __dev_map_flush(). dev_map_flush_old() needs to be cared too. > >> > >> I don't have a problem either way. Jesper, what do you think? > > > > It does sound like my commit 5d053f9da431 ("bpf: devmap prepare xdp > > frames for bulking") introduced this issue. I guess we can add the RCU > > section to xdp_do_flush_map(), and then also verify that the devmap > > (and cpumap) take-down code also have appropriate RCU sections (which > > they should have). > > > > Another requirement for calling .ndo_xdp_xmit is running under NAPI > > protection, is that still satisifed for veth? > > (even when invoked via xdp_do_flush_map()). > > > > virtio_net hits this because of: > xdp_prog = rcu_dereference(rq->xdp_prog); > > in its ndo_xdp_xmit. Scanning ndo_xdp_xmit for other nics does not show > this same check. Is it really required? If so, why don't other drivers > do it? That was my initial thought as well. Can't we just check that vi->xdp_queue_pairs was set in virtnet_xdp_xmit? It is being done just before assigning the XDP prog pointers for rqs. Haven't looked at veth though.