From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751893AbcGMHNg (ORCPT ); Wed, 13 Jul 2016 03:13:36 -0400 Received: from metis.ext.4.pengutronix.de ([92.198.50.35]:50726 "EHLO metis.ext.4.pengutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751232AbcGMHNe (ORCPT ); Wed, 13 Jul 2016 03:13:34 -0400 From: Markus Pargmann To: Pranay Srivastava Cc: "nbd-general@lists.sourceforge.net" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v4 2/5]nbd: fix might_sleep warning on socket shutdown Date: Wed, 13 Jul 2016 09:13:30 +0200 Message-ID: <2779831.bWu4zaVOJ5@adelgunde> User-Agent: KMail/4.14.1 (Linux/4.6.0-0.bpo.1-amd64; KDE/4.14.2; x86_64; ; ) In-Reply-To: References: <1467284524-15676-1-git-send-email-pranjas@gmail.com> <2102261.xdpn3U9KG5@galactica.lan> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart6870931.cSMfGGgRtM"; micalg="pgp-sha256"; protocol="application/pgp-signature" X-SA-Exim-Connect-IP: 2001:67c:670:100:a61f:72ff:fe68:75ba X-SA-Exim-Mail-From: mpa@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-kernel@vger.kernel.org Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --nextPart6870931.cSMfGGgRtM Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="us-ascii" On Sunday 10 July 2016 21:03:05 Pranay Srivastava wrote: > On Sunday, July 10, 2016, Markus Pargmann wrote:= > > Hi, > > > > On 2016 M06 30, Thu 14:02:02 CEST Pranay Kr. Srivastava wrote: > >> spinlocked ranges should be small and not contain calls into huge > >> subfunctions. Fix my mistake and just get the pointer to the socke= t > >> instead of doing everything with spinlock held. > >> > >> Reported-by: Mikulas Patocka > >> Signed-off-by: Markus Pargmann > >> > >> Changelog: > >> Pranay Kr. Srivastava: > >> > >> 1) Use spin_lock instead of irq version for sock_shutdown. > >> > >> 2) Use system work queue to actually trigger the shutdown of > >> socket. This solves the issue when kernel_sendmsg is currently > >> blocked while a timeout occurs. > >> > >> Signed-off-by: Pranay Kr. Srivastava > >> --- > >> drivers/block/nbd.c | 57 > >> +++++++++++++++++++++++++++++++++-------------------- 1 file chang= ed, 36 > >> insertions(+), 21 deletions(-) > >> > >> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c > >> index 766c401..e362d44 100644 > >> --- a/drivers/block/nbd.c > >> +++ b/drivers/block/nbd.c > >> @@ -39,6 +39,7 @@ > >> #include > >> > >> #include > >> +#include > >> > >> struct nbd_device { > >> u32 flags; > >> @@ -69,6 +70,8 @@ struct nbd_device { > >> #if IS_ENABLED(CONFIG_DEBUG_FS) > >> struct dentry *dbg_dir; > >> #endif > >> + /* This is specifically for calling sock_shutdown, for now. = */ > >> + struct work_struct ws_shutdown; > >> }; > >> > >> #if IS_ENABLED(CONFIG_DEBUG_FS) > >> @@ -95,6 +98,8 @@ static int max_part; > >> */ > >> static DEFINE_SPINLOCK(nbd_lock); > >> > >> +static void nbd_ws_func_shutdown(struct work_struct *); > >> + > > > > are you reading all the comments I had?... > > > > At least respond to my comments if you disagree. I still can't see = the > benefit > > of a function signature here if we can avoid it. > > >=20 > That would require some code to be moved. So to avoid those > unnecessary changes it was better to have a prototype. >=20 > It would've pissed you off more if I had tried > to get rid of protoype. Ah I see, thanks. >=20 > >> static inline struct device *nbd_to_dev(struct nbd_device *nbd) > >> { > >> return disk_to_dev(nbd->disk); > >> @@ -172,39 +177,36 @@ static void nbd_end_request(struct nbd_devic= e *nbd, > >> struct request *req) */ > >> static void sock_shutdown(struct nbd_device *nbd) > >> { > >> - spin_lock_irq(&nbd->sock_lock); > >> - > >> - if (!nbd->sock) { > >> - spin_unlock_irq(&nbd->sock_lock); > >> - return; > >> - } > >> + struct socket *sock; > >> > >> - dev_warn(disk_to_dev(nbd->disk), "shutting down socket\n"); > >> - kernel_sock_shutdown(nbd->sock, SHUT_RDWR); > >> - sockfd_put(nbd->sock); > >> + spin_lock(&nbd->sock_lock); > >> + sock =3D nbd->sock; > >> nbd->sock =3D NULL; > >> - spin_unlock_irq(&nbd->sock_lock); > >> + spin_unlock(&nbd->sock_lock); > >> + > >> + if (!sock) > >> + return; > >> > >> del_timer(&nbd->timeout_timer); > >> + dev_warn(disk_to_dev(nbd->disk), "shutting down socket\n"); > >> + kernel_sock_shutdown(sock, SHUT_RDWR); > >> + sockfd_put(sock); > >> } > >> > >> static void nbd_xmit_timeout(unsigned long arg) > >> { > >> struct nbd_device *nbd =3D (struct nbd_device *)arg; > >> - unsigned long flags; > >> > >> if (list_empty(&nbd->queue_head)) > >> return; > >> > >> - spin_lock_irqsave(&nbd->sock_lock, flags); > >> - > >> nbd->timedout =3D true; > >> - > >> - if (nbd->sock) > >> - kernel_sock_shutdown(nbd->sock, SHUT_RDWR); > >> - > >> - spin_unlock_irqrestore(&nbd->sock_lock, flags); > >> - > >> + /* > >> + * Make sure sender thread sees nbd->timedout. > >> + */ > >> + smp_wmb(); > >> + schedule_work(&nbd->ws_shutdown); > >> + wake_up(&nbd->waiting_wq); > >> dev_err(nbd_to_dev(nbd), "Connection timed out, shutting dow= n > >> connection\n"); } > >> > >> @@ -588,7 +590,11 @@ static int nbd_thread_send(void *data) > >> spin_unlock_irq(&nbd->queue_lock); > >> > >> /* handle request */ > >> - nbd_handle_req(nbd, req); > >> + if (nbd->timedout) { > >> + req->errors++; > >> + nbd_end_request(nbd, req); > >> + } else > >> + nbd_handle_req(nbd, req); > > > > I already commented on this in the last patch. This is unrelated to= the > patch. > > If you disagree then please tell me why instead of sending the same= thing > > again. > > >=20 > After trigerring worker thread its > not necessary that socket shutdown > actually was called before we handled > a request. >=20 > So the error would come in > actually later probably. >=20 > So i just wanted to avoid a longer > path for error to be thrown up. > Do correct me if this cant happen. Yes socket shutdown may not have been called when we reach this error handling code. But the timeout timer is usually in the range of seconds= . I would assume that the time between triggering the worker and socket shutdown is within a few milliseconds. We would need to hit exactly thi= s condition which would require a new request to be present. I think this is very unlikely and it would be fine if we have a longer error path there. Am I missing something? Also I just noticed that wake_up(&nbd->waiting_wq) in nbd_xmit_timeout(= ) may not be necessary. In nbd_thread_send(): =09wait_event_interruptible(nbd->waiting_wq, =09=09=09=09 kthread_should_stop() || =09=09=09=09 !list_empty(&nbd->waiting_queue)); =09if (list_empty(&nbd->waiting_queue)) =09=09continue; So wouldn't this wake_up() call simply result in nothing? As soon as sock_shutdown() was called, the receiver thread would exit and close down nbd_thread_send() as well because of kthread_should_stop= (). >=20 > > Also brackets on the else part would be preferred. >=20 > It might trigger checkpatch warning > but I am not 100% sure. Documentation/CodingStyle documents this. See line 168. Best Regards, Markus > > > > Regards, > > > > Markus > > > >> } > >> > >> nbd->task_send =3D NULL; > >> @@ -663,6 +669,7 @@ static void nbd_reset(struct nbd_device *nbd) > >> set_capacity(nbd->disk, 0); > >> nbd->flags =3D 0; > >> nbd->xmit_timeout =3D 0; > >> + INIT_WORK(&nbd->ws_shutdown, nbd_ws_func_shutdown); > >> queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->que= ue); > >> del_timer_sync(&nbd->timeout_timer); > >> } > >> @@ -797,11 +804,11 @@ static int __nbd_ioctl(struct block_device *= bdev, > >> struct nbd_device *nbd, error =3D nbd_thread_recv(nbd, bdev); > >> nbd_dev_dbg_close(nbd); > >> kthread_stop(thread); > >> + sock_shutdown(nbd); > >> > >> mutex_lock(&nbd->tx_lock); > >> nbd->task_recv =3D NULL; > >> > >> - sock_shutdown(nbd); > >> nbd_clear_que(nbd); > >> kill_bdev(bdev); > >> nbd_bdev_reset(bdev); > >> @@ -857,6 +864,14 @@ static const struct block_device_operations > nbd_fops =3D > >> { .compat_ioctl =3D nbd_ioctl, > >> }; > >> > >> +static void nbd_ws_func_shutdown(struct work_struct *ws_nbd) > >> +{ > >> + struct nbd_device *nbd_dev =3D container_of(ws_nbd, struct > nbd_device, > >> + ws_shutdown); > >> + > >> + sock_shutdown(nbd_dev); > >> +} > >> + > >> #if IS_ENABLED(CONFIG_DEBUG_FS) > >> > >> static int nbd_dbg_tasks_show(struct seq_file *s, void *unused) > > > > > > >=20 >=20 =2D-=20 Pengutronix e.K. | = | Industrial Linux Solutions | http://www.pengutronix.de/= | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 = | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-555= 5 | --nextPart6870931.cSMfGGgRtM Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXheoaAAoJENnm3voMNZul/nIP/0bD2u+Ak+oRhqsplXmyWJwM wsFBPkYDqNCzQOFExh22ZsncLZ3JpyIho2z/Sf/AhSHnd5CsE9T5HVTMOE1cqizT 8AgOEz29QZPwGXwgAGJeuUPxUlO3F4XYyNIne3QmwtvBw9U2wa7Cp3adXQ4Nn3BE +QvssxTq6LHU6NrznPNY2jUReUGCXkuv0TNqXo00KX/sv9NVdhjfqF5pQ9kNQjEg CWTTCoKsInDpjRRkleIHONTx6D1IXFZt7eZ8wLARqrGuyuZbj+Wpd4Ir9vV3+yMk rK2LNhJEoD9F4o6W2ovoZSVy3VvqlWfwu2QZbpBfPeJIYQqeUZ8PVe/2OLwNBkZS DJ7nFYe25pF9ccSSxtoye8zT9+p76rnt7x9YdzPmLzlUeHEsZjPU1BNq1jhKI26/ 9Hob1OlQKVdZS21nLXga+JfmdXQSYh6WBlQCRHRm/KNdI6zpspOuo3YXQzYcBAwS ct2gBzFx/ZuioIjZUMQj2BwwckpBJenpKn/EFoD2aBj7J42DIvtiBYbu693AL+6f 6wOLmGbI03yOvFaC+D0hiXQwlpbMJJ0KhgEEyb+zr/WWXpjnwlXL1GiP0P7H7wjH nwWOgNWyUQFjXn24m9RbwXQ2E37yX1XkgIhZwlbC2IgtVt32Fhzw6oxp2wGyY3/3 jsHBtt6O24fCd0RigWwN =ZfXO -----END PGP SIGNATURE----- --nextPart6870931.cSMfGGgRtM--