All of lore.kernel.org
 help / color / mirror / Atom feed
From: Matt Mackall <mpm@selenic.com>
To: Stephen Hemminger <shemminger@linux-foundation.org>
Cc: bridge@lists.linux-foundation.org,
	Flavio Leitner <fleitner@redhat.com>,
	Cong Wang <amwang@redhat.com>,
	Neil Horman <nhorman@tuxdriver.com>,
	netdev@vger.kernel.org, Jay Vosburgh <fubar@us.ibm.com>,
	lkml <linux-kernel@vger.kernel.org>,
	bonding-devel@lists.sourceforge.net,
	Flavio Leitner <fbl@sysclose.org>, Jeff Moyer <jmoyer@redhat.com>,
	Andy Gospodarek <gospo@redhat.com>,
	David Miller <davem@davemloft.net>
Subject: Re: [Bridge] [PATCH] netconsole: queue console messages to send later
Date: Mon, 07 Jun 2010 15:21:31 -0500	[thread overview]
Message-ID: <1275942091.26597.85.camel@calx> (raw)
In-Reply-To: <20100607130015.15555744@nehalam>

On Mon, 2010-06-07 at 13:00 -0700, Stephen Hemminger wrote:
> On Mon, 07 Jun 2010 14:50:48 -0500
> Matt Mackall <mpm@selenic.com> wrote:
> 
> > On Mon, 2010-06-07 at 16:24 -0300, Flavio Leitner wrote:
> > > There are some networking drivers that hold a lock in the
> > > transmit path. Therefore, if a console message is printed
> > > after that, netconsole will push it through the transmit path,
> > > resulting in a deadlock.
> > 
> > This is an ongoing pain we've known about since before introducing the
> > netpoll code to the tree.
> > 
> > My take has always been that any form of queueing is contrary to the
> > goal of netpoll: timely delivery of messages even during machine-killing
> > situations like oopses. There may never be a second chance to deliver
> > the message as the machine may be locked solid. And there may be no
> > other way to get the message out of the box in such situations. Adding
> > queueing is a throwing-the-baby-out-with-the-bathwater fix.
> > 
> > I think Dave agrees with me here, and I believe he's said in the past
> > that drivers trying to print messages in such contexts should be
> > considered buggy.
> > 
> 
> Because it to hard to fix all possible device configurations.
> There should be any way to detect recursion and just drop the message to
> avoid deadlock.

Open to suggestions. The locks in question are driver-internal. There
also may not be any actual recursion taking place:

driver path a takes private lock x
driver path a attempts printk
printk calls into netconsole
netconsole calls into driver path b
driver path b attempts to take lock x -> deadlock

So we can't even try to walk back the stack looking for such nonsense.
Though we could perhaps force queuing of all messages -from- the driver
bound to netconsole. Tricky, and not quite foolproof.

-- 
Mathematics is the supreme nostalgia of our time.



WARNING: multiple messages have this Message-ID (diff)
From: Matt Mackall <mpm@selenic.com>
To: Stephen Hemminger <shemminger@linux-foundation.org>
Cc: Flavio Leitner <fleitner@redhat.com>,
	netdev@vger.kernel.org, David Miller <davem@davemloft.net>,
	Cong Wang <amwang@redhat.com>, Jay Vosburgh <fubar@us.ibm.com>,
	Flavio Leitner <fbl@sysclose.org>,
	Andy Gospodarek <gospo@redhat.com>,
	Neil Horman <nhorman@tuxdriver.com>,
	Jeff Moyer <jmoyer@redhat.com>,
	lkml <linux-kernel@vger.kernel.org>,
	bridge@lists.linux-foundation.org,
	bonding-devel@lists.sourceforge.net
Subject: Re: [PATCH] netconsole: queue console messages to send later
Date: Mon, 07 Jun 2010 15:21:31 -0500	[thread overview]
Message-ID: <1275942091.26597.85.camel@calx> (raw)
In-Reply-To: <20100607130015.15555744@nehalam>

On Mon, 2010-06-07 at 13:00 -0700, Stephen Hemminger wrote:
> On Mon, 07 Jun 2010 14:50:48 -0500
> Matt Mackall <mpm@selenic.com> wrote:
> 
> > On Mon, 2010-06-07 at 16:24 -0300, Flavio Leitner wrote:
> > > There are some networking drivers that hold a lock in the
> > > transmit path. Therefore, if a console message is printed
> > > after that, netconsole will push it through the transmit path,
> > > resulting in a deadlock.
> > 
> > This is an ongoing pain we've known about since before introducing the
> > netpoll code to the tree.
> > 
> > My take has always been that any form of queueing is contrary to the
> > goal of netpoll: timely delivery of messages even during machine-killing
> > situations like oopses. There may never be a second chance to deliver
> > the message as the machine may be locked solid. And there may be no
> > other way to get the message out of the box in such situations. Adding
> > queueing is a throwing-the-baby-out-with-the-bathwater fix.
> > 
> > I think Dave agrees with me here, and I believe he's said in the past
> > that drivers trying to print messages in such contexts should be
> > considered buggy.
> > 
> 
> Because it to hard to fix all possible device configurations.
> There should be any way to detect recursion and just drop the message to
> avoid deadlock.

Open to suggestions. The locks in question are driver-internal. There
also may not be any actual recursion taking place:

driver path a takes private lock x
driver path a attempts printk
printk calls into netconsole
netconsole calls into driver path b
driver path b attempts to take lock x -> deadlock

So we can't even try to walk back the stack looking for such nonsense.
Though we could perhaps force queuing of all messages -from- the driver
bound to netconsole. Tricky, and not quite foolproof.

-- 
Mathematics is the supreme nostalgia of our time.



  reply	other threads:[~2010-06-07 20:21 UTC|newest]

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-05-05  8:11 [Bridge] [v5 Patch 1/3] netpoll: add generic support for bridge and bonding devices Amerigo Wang
2010-05-05  8:11 ` Amerigo Wang
2010-05-05  8:11 ` Amerigo Wang
2010-05-05  8:11 ` [Bridge] [v5 Patch 2/3] bridge: make bridge support netpoll Amerigo Wang
2010-05-05  8:11   ` Amerigo Wang
2010-05-05  8:11   ` Amerigo Wang
2010-05-05  8:11 ` [Bridge] [v5 Patch 3/3] bonding: make bonding " Amerigo Wang
2010-05-05  8:11   ` Amerigo Wang
2010-05-05  8:11   ` Amerigo Wang
2010-05-06  2:05 ` [Bridge] [v5 Patch 1/3] netpoll: add generic support for bridge and bonding devices Matt Mackall
2010-05-06  2:05   ` Matt Mackall
2010-05-06  7:44   ` [Bridge] " David Miller
2010-05-06  7:44     ` David Miller
2010-05-07  3:24     ` [Bridge] " Cong Wang
2010-05-07  3:24       ` Cong Wang
2010-05-27 18:05 ` [Bridge] " Flavio Leitner
2010-05-27 18:05   ` Flavio Leitner
2010-05-27 20:35   ` [Bridge] " David Miller
2010-05-27 20:35     ` David Miller
2010-05-27 21:25     ` [Bridge] " Flavio Leitner
2010-05-27 21:25       ` Flavio Leitner
2010-05-28  2:47   ` [Bridge] " Cong Wang
2010-05-28  2:47     ` Cong Wang
2010-05-28 19:40     ` [Bridge] " Flavio Leitner
2010-05-28 19:40       ` Flavio Leitner
2010-05-31  5:56       ` [Bridge] " Cong Wang
2010-05-31  5:56         ` Cong Wang
2010-05-31 19:08         ` [Bridge] " Flavio Leitner
2010-05-31 19:08           ` Flavio Leitner
2010-06-01  9:57           ` [Bridge] " Cong Wang
2010-06-01  9:57             ` Cong Wang
2010-06-01 18:42             ` [Bridge] " Jay Vosburgh
2010-06-01 18:42               ` Jay Vosburgh
2010-06-02 10:04               ` [Bridge] " Cong Wang
2010-06-02 10:04                 ` Cong Wang
2010-06-04 19:18                 ` [Bridge] " Andy Gospodarek
2010-06-04 19:18                   ` Andy Gospodarek
2010-06-07  9:57                   ` [Bridge] " Cong Wang
2010-06-07  9:57                     ` Cong Wang
2010-06-07 10:01                     ` [Bridge] " David Miller
2010-06-07 10:01                       ` David Miller
2010-06-08  8:36                       ` [Bridge] " Cong Wang
2010-06-08  8:36                         ` Cong Wang
2010-06-07 13:03                     ` [Bridge] " Andy Gospodarek
2010-06-07 13:03                       ` Andy Gospodarek
2010-06-08  8:38                       ` [Bridge] " Cong Wang
2010-06-08  8:38                         ` Cong Wang
2010-06-07 19:24               ` [Bridge] [PATCH] netconsole: queue console messages to send later Flavio Leitner
2010-06-07 19:24                 ` Flavio Leitner
2010-06-07 19:50                 ` [Bridge] " Matt Mackall
2010-06-07 19:50                   ` Matt Mackall
2010-06-07 20:00                   ` [Bridge] " Stephen Hemminger
2010-06-07 20:00                     ` Stephen Hemminger
2010-06-07 20:21                     ` Matt Mackall [this message]
2010-06-07 20:21                       ` Matt Mackall
2010-06-07 23:52                       ` [Bridge] " David Miller
2010-06-07 23:52                         ` David Miller
2010-06-07 23:50                 ` [Bridge] " David Miller
2010-06-07 23:50                   ` David Miller
2010-06-08  0:37                   ` [Bridge] " Flavio Leitner
2010-06-08  0:37                     ` Flavio Leitner
2010-06-08  8:59                     ` [Bridge] " Cong Wang
2010-06-08  8:59                       ` Cong Wang
2010-05-28  8:16   ` [Bridge] [v5 Patch 1/3] netpoll: add generic support for bridge and bonding devices Cong Wang
2010-05-28  8:16     ` Cong Wang
2010-05-28 20:42     ` [Bridge] " Flavio Leitner
2010-05-28 20:42       ` Flavio Leitner
2010-05-28 21:03       ` [Bridge] " Jay Vosburgh
2010-05-28 21:03         ` Jay Vosburgh
2010-05-31  5:29         ` [Bridge] " Cong Wang
2010-05-31  5:29           ` Cong Wang
2010-05-31  5:37           ` [Bridge] " Cong Wang
2010-05-31  5:37             ` Cong Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1275942091.26597.85.camel@calx \
    --to=mpm@selenic.com \
    --cc=amwang@redhat.com \
    --cc=bonding-devel@lists.sourceforge.net \
    --cc=bridge@lists.linux-foundation.org \
    --cc=davem@davemloft.net \
    --cc=fbl@sysclose.org \
    --cc=fleitner@redhat.com \
    --cc=fubar@us.ibm.com \
    --cc=gospo@redhat.com \
    --cc=jmoyer@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=nhorman@tuxdriver.com \
    --cc=shemminger@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.