From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mohammed Gamal Subject: Re: [PATCH] hv_netvsc: Make sure out channel is fully opened on send Date: Thu, 27 Sep 2018 12:31:21 +0200 Message-ID: <1538044281.19334.4.camel@redhat.com> References: <1537979659-26979-1-git-send-email-mgamal@redhat.com> <1538038625.19334.2.camel@redhat.com> <20180927122355.470df119@shemminger-XPS-13-9360> Reply-To: mgamal@redhat.com Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Cc: Haiyang Zhang , Stephen Hemminger , "netdev@vger.kernel.org" , "otubo@redhat.com" , "linux-kernel@vger.kernel.org" , "devel@linuxdriverproject.org" , vkuznets To: Stephen Hemminger Return-path: In-Reply-To: <20180927122355.470df119@shemminger-XPS-13-9360> Sender: linux-kernel-owner@vger.kernel.org List-Id: netdev.vger.kernel.org On Thu, 2018-09-27 at 12:23 +0200, Stephen Hemminger wrote: > On Thu, 27 Sep 2018 10:57:05 +0200 > Mohammed Gamal wrote: > > > On Wed, 2018-09-26 at 17:13 +0000, Haiyang Zhang wrote: > > > > -----Original Message----- > > > > From: Mohammed Gamal > > > > Sent: Wednesday, September 26, 2018 12:34 PM > > > > To: Stephen Hemminger ; netdev@vger.ker > > > > nel. > > > > org > > > > Cc: KY Srinivasan ; Haiyang Zhang > > > > ; vkuznets ; > > > > otubo@redhat.com; cavery ; linux- > > > > kernel@vger.kernel.org; devel@linuxdriverproject.org; Mohammed > > > > Gamal > > > > > > > > Subject: [PATCH] hv_netvsc: Make sure out channel is fully > > > > opened > > > > on send > > > > > > > > Dring high network traffic changes to network interface > > > > parameters > > > > such as > > > > number of channels or MTU can cause a kernel panic with a NULL > > > > pointer > > > > dereference. This is due to netvsc_device_remove() being called > > > > and > > > > deallocating the channel ring buffers, which can then be > > > > accessed > > > > by > > > > netvsc_send_pkt() before they're allocated on calling > > > > netvsc_device_add() > > > > > > > > The patch fixes this problem by checking the channel state and > > > > returning > > > > ENODEV if not yet opened. We also move the call to > > > > hv_ringbuf_avail_percent() > > > > which may access the uninitialized ring buffer. > > > > > > > > Signed-off-by: Mohammed Gamal > > > > --- > > > >  drivers/net/hyperv/netvsc.c | 7 ++++++- > > > >  1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/net/hyperv/netvsc.c > > > > b/drivers/net/hyperv/netvsc.c index > > > > fe01e14..75f1b31 100644 > > > > --- a/drivers/net/hyperv/netvsc.c > > > > +++ b/drivers/net/hyperv/netvsc.c > > > > @@ -825,7 +825,12 @@ static inline int netvsc_send_pkt( > > > >   struct netdev_queue *txq = netdev_get_tx_queue(ndev, > > > > packet->q_idx); > > > >   u64 req_id; > > > >   int ret; > > > > - u32 ring_avail = > > > > hv_get_avail_to_write_percent(&out_channel-   > > > > > outbound);   > > > > > > > > + u32 ring_avail; > > > > + > > > > + if (out_channel->state != CHANNEL_OPENED_STATE) > > > > + return -ENODEV; > > > > + > > > > + ring_avail = > > > > hv_get_avail_to_write_percent(&out_channel-   > > > > > outbound);   > > > > > > When you reproducing the NULL ptr panic, does your kernel include > > > the > > > following patch? > > > hv_netvsc: common detach logic > > > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.g > > > it/c > > > ommit/?id=7b2ee50c0cd513a176a26a71f2989facdd75bfea > > >    > > > > Yes it is included. And the commit did reduce the occurrence of > > this > > race condition, but it still nevertheless occurs albeit rarely. > > > > > We call netif_tx_disable(ndev) and netif_device_detach(ndev) > > > before > > > doing the changes  > > > on MTU or #channels. So there should be no call to start_xmit() > > > when > > > channel is not ready. > > > > > > If you see the check for CHANNEL_OPENED_STATE is still necessary > > > on > > > upstream kernel (including  > > > the patch " common detach logic "), we should debug further on > > > the > > > code and find out the  > > > root cause. > > > > > > Thanks, > > > - Haiyang > > >    > > > > _______________________________________________ > > devel mailing list > > devel@linuxdriverproject.org > > http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev- > > devel > > Is there some workload, that can be used to reproduce this? > The stress test from Vitaly with changing parameters while running > network traffic > passes now. > > Can you reproduce this with the upstream current kernel? > > Adding the check in start xmit is still racy, and won't cure the > problem. > > Another solution would be to add a grace period in the netvsc detach > logic. > Steps to reproduce are listed here: https://bugzilla.redhat.com/show_bug.cgi?id=1632653 We've also managed to reproduce the same issue upstream. It's more likely to be reproduced on Windows 2012R2 than 2016. Regards, Mohammed