From: Michael Breuer <mbreuer@majjas.com>
To: Stephen Hemminger <shemminger@vyatta.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
"Berck E. Nash" <flyboy@gmail.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
netdev@vger.kernel.org
Subject: Re: sky2 panic in 2.6.32.1 under load (tty NULL write)
Date: Fri, 01 Jan 2010 14:26:53 -0500 [thread overview]
Message-ID: <4B3E4C7D.2040802@majjas.com> (raw)
In-Reply-To: <4B3E33EC.1060703@majjas.com>
Running with TTY_DEBUG_HANGUP:
On boot - lots of stuff opening and closing tty1. Once up and stable
(runlevel 3), I did initctl stop tty1, and then initctl start tty1. This
is what was logged:
Jan 1 14:03:21 mail root: close mingetty1
Jan 1 14:03:21 mail kernel: tty_release_dev of tty1 (tty count=6)...
Jan 1 14:03:21 mail kernel: tty_release_dev of tty1 (tty count=5)...
Jan 1 14:03:21 mail kernel: tty1 vhangup...
Jan 1 14:03:21 mail init: tty1 main process (6031) killed by TERM signal
Jan 1 14:03:36 mail root: start mingetty1
Jan 1 14:03:36 mail kernel: error attempted to write to tty [0x(null)]
= NULL
Jan 1 14:03:36 mail kernel: opening tty1...
Jan 1 14:03:36 mail kernel: tty1 vhangup...
Jan 1 14:03:36 mail kernel: tty_release_dev of tty1 (tty count=5)...
Start/stop of mingetty on other ttys does not get the NULL write error.
I am always getting the NULL error when starting mingetty on tty1.
For example:
Jan 1 14:04:11 mail root: restart mingetty2
Jan 1 14:04:11 mail init: tty2 main process (6029) killed by TERM signal
Jan 1 14:04:11 mail kernel: tty_release_dev of tty7 (tty count=2)...
Jan 1 14:04:11 mail kernel: tty_release_dev of tty2 (tty count=1)...
Jan 1 14:04:11 mail kernel: freeing tty structure...
Jan 1 14:04:11 mail kernel: opening tty2...
Jan 1 14:04:11 mail kernel: tty2 vhangup...
Jan 1 14:04:11 mail kernel: tty_release_dev of tty2 (tty count=1)...
Jan 1 14:04:11 mail kernel: freeing tty structure...
While starting/stopping tty2 and X was coming up, I also got this:
Jan 1 14:04:11 mail root: restart mingetty2
Jan 1 14:04:11 mail init: tty2 main process (6029) killed by TERM signal
Jan 1 14:04:11 mail kernel: tty_release_dev of tty7 (tty count=2)...
Jan 1 14:04:11 mail kernel: tty_release_dev of tty2 (tty count=1)...
Jan 1 14:04:11 mail kernel: freeing tty structure...
Jan 1 14:04:11 mail kernel: opening tty2...
Jan 1 14:04:11 mail kernel: tty2 vhangup...
Jan 1 14:04:11 mail kernel: tty_release_dev of tty2 (tty count=1)...
Jan 1 14:04:11 mail kernel: freeing tty structure...
Jan 1 14:04:17 mail gnome-session[6832]: WARNING: Could not parse
desktop file
/home/mbreuer/.config/autostart/xfce4-tips-autostart.desktop: Key file
does not have key 'Name'
Jan 1 14:04:17 mail gnome-session[6832]: WARNING: could not read
/home/mbreuer/.config/autostart/xfce4-tips-autostart.desktop
Jan 1 14:04:19 mail pulseaudio[7002]: pid.c: Stale PID file, overwriting.
Jan 1 14:04:20 mail kernel: opening tty2...
Jan 1 14:04:20 mail kernel: opening pts2...
Jan 1 14:04:20 mail kernel: opening pts2...
Jan 1 14:04:20 mail kernel: tty_release_dev of pts2 (tty count=3)...
Jan 1 14:04:20 mail kernel: opening pts2...
Jan 1 14:04:20 mail kernel: tty_release_dev of pts2 (tty count=3)...
Jan 1 14:04:20 mail kernel: opening pts3...
Jan 1 14:04:20 mail kernel: opening pts3...
Jan 1 14:04:20 mail kernel: tty_release_dev of pts3 (tty count=3)...
Jan 1 14:04:20 mail kernel: opening pts3...
Jan 1 14:04:20 mail kernel: tty_release_dev of pts3 (tty count=3)...
Jan 1 14:04:20 mail kernel: opening pts4...
Jan 1 14:04:20 mail kernel: opening pts4...
Jan 1 14:04:20 mail kernel: tty_release_dev of pts4 (tty count=3)...
Jan 1 14:04:20 mail kernel: opening pts4...
Jan 1 14:04:21 mail kernel: tty_release_dev of pts4 (tty count=3)...
Jan 1 14:04:21 mail kernel: DMA-API: debugging out of memory - disabling
Jan 1 14:04:23 mail pulseaudio[7417]: pid.c: Daemon already running.
Jan 1 14:04:24 mail kernel: sky2 0000:06:00.0: error interrupt status=0x8
Jan 1 14:04:24 mail kernel: sky2 Tx ring pending=45...52 report=45 done=47
Jan 1 14:04:24 mail kernel: 44: 0x0: 0xf26a40be(9014)
Jan 1 14:04:24 mail kernel: 46: 0x0: 0xf26a00be(9014)
Jan 1 14:04:24 mail kernel: 48: 0x0: 0xf269c0be(9014)
Jan 1 14:04:24 mail kernel: 50: 0x0: 0xf26980be(9014)
Jan 1 14:04:24 mail kernel: sky2 0000:06:00.0: error interrupt status=0x8
Jan 1 14:04:24 mail kernel: sky2 Tx ring pending=47...52 report=47 done=49
Jan 1 14:04:24 mail kernel: 46: 0x0: 0xf26a00be(9014)
Jan 1 14:04:24 mail kernel: 48: 0x0: 0xf269c0be(9014)
Jan 1 14:04:24 mail kernel: 50: 0x0: 0xf26980be(9014)
Note the sky2 errors (this time not under load).
It looks like there is something bad going on with the tty driver.
On 01/01/2010 12:42 PM, Michael Breuer wrote:
> More results... Seems that this is related to the tty (or maybe vt or
> maybe console) driver. I had disabled mingetty on tty1 as a workaround
> to a Fedora/KDE issue. When I reenabled it, I was then able to
> recreate this problem (sky2) in runlevel 3.
>
> Retesting, it seems that the following conditions must be met in order
> for the interrupt errors to happen:
>
> 1) mingetty running on tty1 before the start of the test run
> 2) High sustained incoming load - 25MB/sec or more (as reported by
> nethogs)
> 3) I've only been able to recreate this using samba & a Windows 7
> backup - but there could be other triggers
>
> As it happens, much of my other data all came down to mingetty
> starting or not starting on tty1 at specific points of test
> preparation. At this point I can recreate 100% of the time if I ensure
> that there is a mingetty running on tty1, regardless of runlevel; and
> I can not recreate 100% of the time when there is no mingetty running
> on tty1 at the start of the test. The results do not change if I start
> or stop mingetty on tty1 during the test. It's possible that X running
> on tty1 may also trigger the issue, but I haven't tested that. Note
> that running mingetty on the other tty's does not affect the outcome
> of the test. The test is also unaffected if I chvt to or away from
> tty1 before the test. I do not have to be logged in at the console (on
> any vt) to trigger the issue.
>
> In a nutshell, to recreate: make sure there is a mingetty running on
> tty1, blast traffic into the system on eth0 (sky2).
>
>
>
>
> On 12/31/2009 1:09 PM, Michael Breuer wrote:
>> Did some more digging today... Haven't nailed it, but there's
>> something going on with X and tty...
>>
>> Among other things, when I telinit 3 && telinit 5 the tty keeps
>> switching between 7 and 8 (nothing else running on either tty). It
>> would appear that somehow the tty deallocation isn't complete when X
>> restarts. Also, X grabbing a tty seems to be a requisite step in
>> recreating the sky2 issue.
>>
>> On 12/30/2009 1:10 PM, Stephen Hemminger wrote:
>>> On Wed, 30 Dec 2009 10:40:56 -0500
>>> Michael Breuer<mbreuer@majjas.com> wrote:
>>>
>>>> Dec 30 10:37:14 mail kernel: sky2 0000:06:00.0: error interrupt
>>>> status=0x40000008
>>>> Dec 30 10:37:14 mail kernel: sky2 Tx ring pending=31...40 report=35
>>>> done=35
>>>> Dec 30 10:37:14 mail kernel: 30: 0x0: 0xeb50c0ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 32: 0x0: 0xeb5080ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 34: 0x0: 0xeb5040ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 36: 0x0: 0xeb5000ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 38: 0x0: 0xeb4fc0ca(9014)
>>> You must be using 9K MTU. Do you see the problem with smaller
>>> MTU?
>>>
>>>> Dec 30 10:37:14 mail kernel: sky2 0000:06:00.0: error interrupt
>>>> status=0x40000008
>>>> Dec 30 10:37:14 mail kernel: sky2 Tx ring pending=87...92 report=89
>>>> done=91
>>>> Dec 30 10:37:14 mail kernel: 86: 0x0: 0xeafd80ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 88: 0x0: 0xeafd40ca(9014)
>>>> Dec 30 10:37:14 mail kernel: 90: 0x0: 0xeafd00ca(9014)
>>>> Dec 30 10:37:15 mail kernel: sky2 0000:06:00.0: error interrupt
>>>> status=0x8
>>>> Dec 30 10:37:15 mail kernel: sky2 Tx ring pending=39...42 report=39
>>>> done=41
>>>> Dec 30 10:37:15 mail kernel: 38: 0x0: 0xe5fec0ca(9014)
>>>> Dec 30 10:37:15 mail kernel: 40: 0x0: 0xe5fe80ca(9014)
>>>> Dec 30 10:37:15 mail kernel: sky2 0000:06:00.0: error interrupt
>>>> status=0x40000008
>>>> Dec 30 10:37:15 mail kernel: sky2 Tx ring pending=117...124 report=119
>>>> done=119
>>>> Dec 30 10:37:15 mail kernel: 116: 0x0: 0xe5d6c0ca(9014)
>>>> Dec 30 10:37:15 mail kernel: 118: 0x0: 0xe5d680ca(9014)
>>>> Dec 30 10:37:15 mail kernel: 120: 0x0: 0xe5d660ca(5938)
>>>> Dec 30 10:37:15 mail kernel: 122: 0x0: 0xe5d640ca(4213)
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
next prev parent reply other threads:[~2010-01-01 19:27 UTC|newest]
Thread overview: 145+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-12-21 23:52 sky2 panic in 2.6.32.1 under load Berck E. Nash
2009-12-22 0:09 ` Michael Breuer
2009-12-22 18:50 ` Michael Breuer
2009-12-23 22:54 ` sky2 panic in 2.6.32.1 under load (new oops) Michael Breuer
2009-12-24 7:01 ` Andrew Morton
2009-12-24 19:18 ` Michael Breuer
2009-12-24 22:27 ` Stephen Hemminger
2009-12-25 16:28 ` Michael Breuer
2009-12-25 23:22 ` Stephen Hemminger
2009-12-26 3:23 ` Michael Breuer
2009-12-26 17:57 ` Stephen Hemminger
2009-12-26 20:37 ` Michael Breuer
2009-12-26 22:05 ` [PATCH] sky2: make sure ethernet header is in transmit skb Stephen Hemminger
2009-12-27 3:44 ` David Miller
2009-12-27 4:11 ` David Miller
2010-01-04 5:32 ` David Miller
2010-01-04 16:40 ` Stephen Hemminger
2010-01-04 17:02 ` Michael Breuer
2010-01-05 23:07 ` [PATCH] af_packet: Don't use skb after dev_queue_xmit() Jarek Poplawski
2010-01-05 23:16 ` Michael Breuer
2010-01-05 23:29 ` Jarek Poplawski
2010-01-06 2:36 ` Michael Breuer
2010-01-06 7:22 ` Jarek Poplawski
2010-01-06 9:15 ` [PATCH alt.2] " Jarek Poplawski
2010-01-06 14:49 ` Stephen Hemminger
2010-01-06 19:40 ` Jarek Poplawski
2010-01-06 19:49 ` [PATCH] " Michael Breuer
2010-01-06 20:22 ` Jarek Poplawski
2010-01-06 20:33 ` Michael Breuer
2010-01-06 21:09 ` Jarek Poplawski
2010-01-06 21:32 ` Michael Breuer
2010-01-06 21:10 ` Stephen Hemminger
2010-01-06 21:20 ` Michael Breuer
2010-01-06 23:26 ` Michael Breuer
2010-01-07 2:42 ` Michael Breuer
2010-01-07 4:00 ` Michael Breuer
2010-01-07 4:53 ` Stephen Hemminger
2010-01-07 5:10 ` Michael Breuer
2010-01-07 5:32 ` Michael Breuer
2010-01-07 5:54 ` Michael Breuer
2010-01-07 7:20 ` Michael Breuer
2010-01-07 7:47 ` Jarek Poplawski
2010-01-07 7:55 ` Michael Breuer
2010-01-07 8:21 ` Jarek Poplawski
2010-01-07 15:03 ` Michael Breuer
2010-01-07 17:56 ` Jarek Poplawski
2010-01-07 18:17 ` Jarek Poplawski
2010-01-07 15:05 ` Michael Breuer
2010-01-07 18:01 ` Jarek Poplawski
2010-01-07 18:19 ` Michael Breuer
2010-01-07 18:35 ` Jarek Poplawski
2010-01-07 18:40 ` Michael Breuer
2010-01-07 18:43 ` Michael Breuer
2010-01-07 18:50 ` Jarek Poplawski
2010-01-07 19:36 ` Jarek Poplawski
2010-01-07 19:55 ` Michael Breuer
2010-01-07 20:22 ` Jarek Poplawski
2010-01-07 23:11 ` Michael Breuer
2010-01-08 7:45 ` Jarek Poplawski
2010-01-08 16:40 ` Michael Breuer
2010-01-08 21:29 ` Jarek Poplawski
2010-01-08 21:48 ` Michael Breuer
2010-01-08 22:02 ` Jarek Poplawski
2010-01-09 4:45 ` Michael Breuer
2010-01-09 5:44 ` Michael Breuer
2010-01-09 12:28 ` Jarek Poplawski
2010-01-09 18:34 ` Michael Breuer
2010-01-13 20:39 ` Michael Breuer
2010-01-13 21:09 ` Jarek Poplawski
2010-01-13 21:16 ` Michael Breuer
2010-01-13 21:34 ` Jarek Poplawski
2010-01-17 16:26 ` Michael Breuer
2010-01-17 22:17 ` Jarek Poplawski
2010-01-17 22:34 ` Michael Breuer
2010-01-17 23:05 ` Jarek Poplawski
2010-01-17 23:15 ` Michael Breuer
2010-01-18 7:30 ` Jarek Poplawski
2010-01-18 16:29 ` Michael Breuer
2010-01-18 20:46 ` Jarek Poplawski
2010-01-18 20:56 ` Michael Breuer
2010-01-18 21:00 ` Stephen Hemminger
2010-01-18 21:06 ` Jarek Poplawski
2010-01-18 21:24 ` Michael Breuer
2010-01-18 21:50 ` Jarek Poplawski
2010-01-18 21:25 ` Jarek Poplawski
2010-01-18 21:39 ` Michael Breuer
2010-01-18 22:08 ` Jarek Poplawski
2010-01-18 22:17 ` Jarek Poplawski
2010-01-18 22:47 ` Michael Breuer
2010-01-19 5:46 ` Michael Breuer
2010-01-19 8:41 ` Jarek Poplawski
2010-01-19 15:28 ` Michael Breuer
2010-01-21 19:48 ` Michael Breuer
2010-01-19 10:47 ` Jarek Poplawski
2010-01-19 15:47 ` Michael Breuer
2010-01-19 19:59 ` Jarek Poplawski
2010-01-19 20:06 ` Michael Breuer
2010-01-19 20:29 ` Jarek Poplawski
2010-01-19 22:45 ` Jarek Poplawski
2010-01-20 1:01 ` Michael Breuer
2010-01-20 1:10 ` Stephen Hemminger
2010-01-21 16:14 ` Stefan Richter
2010-01-21 16:50 ` Stefan Richter
2010-01-18 22:25 ` Michael Breuer
2010-01-18 22:40 ` Jarek Poplawski
2009-12-27 17:03 ` sky2 panic in 2.6.32.1 under load (new oops) Michael Breuer
2009-12-27 18:22 ` Stephen Hemminger
2009-12-27 19:39 ` Michael Breuer
2009-12-29 17:30 ` Stephen Hemminger
2009-12-29 17:39 ` Michael Breuer
2009-12-29 18:38 ` Michael Breuer
2009-12-29 18:54 ` Michael Breuer
2009-12-29 19:49 ` Stephen Hemminger
2009-12-29 20:41 ` Michael Breuer
2009-12-30 7:23 ` Michael Breuer
2009-12-30 7:58 ` Stephen Hemminger
2009-12-30 17:49 ` Michael Breuer
2009-12-30 19:15 ` audit.c skb - tty race condition - was " Michael Breuer
2009-12-30 20:44 ` Michael Breuer
2009-12-30 21:15 ` Michael Breuer
2009-12-30 21:21 ` Michael Breuer
2009-12-30 7:59 ` Stephen Hemminger
2009-12-30 15:40 ` Michael Breuer
2009-12-30 18:10 ` Stephen Hemminger
2009-12-30 18:37 ` Michael Breuer
2009-12-31 18:09 ` Michael Breuer
2009-12-31 18:24 ` Stephen Hemminger
2010-01-01 17:42 ` Michael Breuer
2010-01-01 19:26 ` Michael Breuer [this message]
2010-01-01 20:34 ` sky2 panic in 2.6.32.1 under load (tty NULL write) Michael Breuer
2010-01-02 21:42 ` Michael Breuer
2009-12-29 19:15 ` sky2 panic in 2.6.32.1 under load (new oops) Jarek Poplawski
2009-12-29 19:20 ` Michael Breuer
2009-12-30 8:07 ` Stephen Hemminger
2009-12-30 15:36 ` Michael Breuer
2009-12-22 0:52 ` sky2 panic in 2.6.32.1 under load Daniel Hazelton
2009-12-24 6:58 ` Andrew Morton
2009-12-24 16:03 ` Berck Nash
2009-12-24 16:28 ` Daniel Hazelton
2009-12-24 22:21 ` Stephen Hemminger
2009-12-24 22:42 ` Michael Breuer
2009-12-25 0:06 ` Daniel Hazelton
2009-12-24 16:10 ` Michael Breuer
2009-12-24 16:16 ` Berck Nash
2009-12-24 16:26 ` Michael Breuer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B3E4C7D.2040802@majjas.com \
--to=mbreuer@majjas.com \
--cc=akpm@linux-foundation.org \
--cc=flyboy@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=shemminger@vyatta.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox