[linux-dvb] Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[linux-dvb] Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Brendon Higgins]

[-- Attachment #1.1: Type: text/plain, Size: 1857 bytes --]

Hi,

I've been coming up against a problem that seems to be with the DVB drivers 
that occurs when a program using them, usually VDR in my case, terminates 
uncleanly (segfault, general protection fault). Linux 2.6.25 doesn't have this 
problem, but 2.6.26, 2.6.27, and 2.6.28 do, though 2.6.28 manifests slightly 
differently to the others. I'll focus on what 2.6.28 does. I have a DViCO 
FusionHDTV DVB-T Plus, running on an amd64 Debian Testing (mostly) dual-core 
machine.

After VDR crashes it attemps to restart itself. This was fine on 2.6.25, but on 
later kernels the crash seems to leave the device in some unusable state, 
where no program can subsequently use it - the device files (/dev/dvb) no 
longer exist. (In 2.6.2[67], the files existed, but accessing 
/dev/dvb/adapter0/dvr0 resulted in "No such device".) I had figured out that in 
order to get the device working again it is necessary to "rmmod cx88_dvb 
cx8802; modprobe cx88_dvb". This worked in 2.6.2[67] without trouble, but in 
2.6.28, it's as if the cx88_dvb module gets lost somehow. It doesn't appear in 
lsmod, however:
phi:~# modprobe cx88_dvb
FATAL: Error inserting cx88_dvb 
(/lib/modules/2.6.28/kernel/drivers/media/video/cx88/cx88-dvb.ko): No such 
device
phi:~# rmmod cx88_dvb cx8802
ERROR: Module cx88_dvb does not exist in /proc/modules
phi:~# modprobe cx88_dvb

Note that probing it works only *after* cx8802 was unloaded. As before, now 
the device is accessible.

So it seems something is wrong in the cx8802 module. Something is not being 
cleaned up after a userspace program crashes while using it, leaving the DVB 
system in a broken state.

I'd very much like this to not be the case, since on my system a VDR crash is 
somewhat inevitable, and the automatic restart *was* very handy, back in 
2.6.25.

Peace,
Brendon


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
linux-dvb mailing list
linux-dvb@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb

Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Brendon Higgins]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

Hi,

I wrote to linux-dvb (2009-01-03 12:00 pm):
> I've been coming up against a problem that seems to be with the DVB drivers
> that occurs when a program using them, usually VDR in my case, terminates
> uncleanly (segfault, general protection fault). Linux 2.6.25 doesn't have
> this problem, but 2.6.26, 2.6.27, and 2.6.28 do, though 2.6.28 manifests
> slightly differently to the others. I'll focus on what 2.6.28 does. I have
> a DViCO FusionHDTV DVB-T Plus, running on an amd64 Debian Testing (mostly)
> dual-core machine.
>
> After VDR crashes it attemps to restart itself. This was fine on 2.6.25,
> but on later kernels the crash seems to leave the device in some unusable
> state, where no program can subsequently use it - the device files
> (/dev/dvb) no longer exist. (In 2.6.2[67], the files existed, but accessing
> /dev/dvb/adapter0/dvr0 resulted in "No such device".) I had figured out
> that in order to get the device working again it is necessary to "rmmod
> cx88_dvb cx8802; modprobe cx88_dvb". This worked in 2.6.2[67] without
> trouble, but in 2.6.28, it's as if the cx88_dvb module gets lost somehow.
> It doesn't appear in lsmod, however:
> phi:~# modprobe cx88_dvb
> FATAL: Error inserting cx88_dvb
> (/lib/modules/2.6.28/kernel/drivers/media/video/cx88/cx88-dvb.ko): No such
> device
> phi:~# rmmod cx88_dvb cx8802
> ERROR: Module cx88_dvb does not exist in /proc/modules
> phi:~# modprobe cx88_dvb
>
> Note that probing it works only *after* cx8802 was unloaded. As before, now
> the device is accessible.
>
> So it seems something is wrong in the cx8802 module. Something is not being
> cleaned up after a userspace program crashes while using it, leaving the
> DVB system in a broken state.
>
> I'd very much like this to not be the case, since on my system a VDR crash
> is somewhat inevitable, and the automatic restart *was* very handy, back in
> 2.6.25.

Deafening silence. Does nobody have a clue? Or care? I just noticed I posted 
to the linux-dvb list which has been deprecated, so I'm quoting it here in 
its entirety in case relevent people missed it. Is there anything else I can 
do?

Peace,
Brendon

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Andy Walls]

On Mon, 2009-01-19 at 13:37 +1000, Brendon Higgins wrote:
> Hi,
> 
> I wrote to linux-dvb (2009-01-03 12:00 pm):
> > I've been coming up against a problem that seems to be with the DVB drivers
> > that occurs when a program using them, usually VDR in my case, terminates
> > uncleanly (segfault, general protection fault). Linux 2.6.25 doesn't have
> > this problem, but 2.6.26, 2.6.27, and 2.6.28 do, though 2.6.28 manifests
> > slightly differently to the others. I'll focus on what 2.6.28 does. I have
> > a DViCO FusionHDTV DVB-T Plus, running on an amd64 Debian Testing (mostly)
> > dual-core machine.
> >
> > After VDR crashes it attemps to restart itself. This was fine on 2.6.25,
> > but on later kernels the crash seems to leave the device in some unusable
> > state, where no program can subsequently use it - the device files
> > (/dev/dvb) no longer exist. (In 2.6.2[67], the files existed, but accessing
> > /dev/dvb/adapter0/dvr0 resulted in "No such device".) I had figured out
> > that in order to get the device working again it is necessary to "rmmod
> > cx88_dvb cx8802; modprobe cx88_dvb". This worked in 2.6.2[67] without
> > trouble, but in 2.6.28, it's as if the cx88_dvb module gets lost somehow.
> > It doesn't appear in lsmod, however:
> > phi:~# modprobe cx88_dvb
> > FATAL: Error inserting cx88_dvb
> > (/lib/modules/2.6.28/kernel/drivers/media/video/cx88/cx88-dvb.ko): No such
> > device
> > phi:~# rmmod cx88_dvb cx8802
> > ERROR: Module cx88_dvb does not exist in /proc/modules
> > phi:~# modprobe cx88_dvb
> >
> > Note that probing it works only *after* cx8802 was unloaded. As before, now
> > the device is accessible.
> >
> > So it seems something is wrong in the cx8802 module. Something is not being
> > cleaned up after a userspace program crashes while using it, leaving the
> > DVB system in a broken state.
> >
> > I'd very much like this to not be the case, since on my system a VDR crash
> > is somewhat inevitable, and the automatic restart *was* very handy, back in
> > 2.6.25.
> 
> Deafening silence. Does nobody have a clue?

Well, you have the clues actually; they're in your log files.

At the time of the crash, what shows up in dmesg, /var/log/messages, and
any log that VDR creates?  

Since modprobe after the crash failed with -ENODEV ["...cx88-dvb.ko): No
such device], a quick grep through cx88-dvb.c in the source shows that
that can only happen in a few places.  It should be easy to spot in the
logs.  Adding an

	options cx88_dvb debug=1

line to /etc/modprobe.conf would make things easier to see in the logs
as well.

Also what is the source of your cx88 driver: Debian Testing or the
v4l-dvb repo?


> Or care?

The issue is usually not one of caring, but one of having
time, the specific hardware, and ability to reporduce the problem.

Regards,
Andy


>  I just noticed I posted 
> to the linux-dvb list which has been deprecated, so I'm quoting it here in 
> its entirety in case relevent people missed it. Is there anything else I can 
> do?
> 
> Peace,
> Brendon

Re: Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Brendon Higgins]

[-- Attachment #1: Type: text/plain, Size: 5819 bytes --]

Andy Walls wrote (Monday 19 January 2009):
> On Mon, 2009-01-19 at 13:37 +1000, Brendon Higgins wrote:
> > > [snip]
>
> Well, you have the clues actually; they're in your log files.

Thanks Andy for prompting me to look at things again. I've discovered 
something interesting about the way runvdr, a script which tries to keep vdr 
alive (might be Debian custom, I'm not sure), interacts with the modules. When 
vdr crashes, runvdr attempts to remove and re-load the dvb modules before 
restarting vdr. Presumably this is a workaround for any modules left in a bad 
state.

Interestingly though, the script doesn't take the module dependency tree into 
account, only the first level branches, and so most of its rmmod commands fail 
since the modules are in use by other modules. In my case it attempts to 
remove videobuf_dvb, cx88_dvb, and dvb_core, in that order. Only rmmod 
cx88_dvb succeeds.

Now the really interesting part: soon after that, runvdr tries to modprobe all 
those modules back in again. Trying to modprobe videobuf_dvb and dvb_core does 
nothing since they weren't unloaded. But trying to modprobe cx88_dvb fails: 
"FATAL: Error inserting cx88_dvb 
(/lib/modules/2.6.28/kernel/drivers/media/video/cx88/cx88-dvb.ko): No such 
device"

Only after "rmmod cx8802" can cx88_dvb be loaded successfully.

Finally I've found a way to reproduce the bug on command! Hurrah!

Summary procedure, starting with a working dvb:
1) rmmod cx88_dvb
2) modprobe cx88_dvb
Error: No such device.
3) rmmod cx8802
4) modprobe cx88_dvb
Success (and cx8802 is pulled in automatically)

So it seems there might be some sort of module interdependency not being taken 
care of. I'll try to report the runvdr tree problem to the relevant place 
later.

> At the time of the crash, what shows up in dmesg, /var/log/messages, and
> any log that VDR creates?

Here's what's in /var/log/syslog when I perform the above procedure (with my 
best-guess labels as to when I did what):

rmmod cx88_dvb:
Jan 19 19:24:27 phi kernel: [15162.725955] cx88/2: unregistering cx8802 
driver, type: dvb access: shared
Jan 19 19:24:27 phi kernel: [15162.725967] cx88[0]/2: subsystem: 18ac:db10, 
board: DViCO FusionHDTV DVB-T Plus [card=21]
Jan 19 19:24:27 phi kernel: [15162.725972] cx88[0]/2-dvb: cx8802_dvb_remove

modprobe cx88_dvb:
Jan 19 19:24:32 phi kernel: [15167.399736] cx88/2: cx2388x dvb driver version 
0.0.6 loaded
Jan 19 19:24:32 phi kernel: [15167.399744] cx88/2: registering cx8802 driver, 
type: dvb access: shared
Jan 19 19:24:32 phi kernel: [15167.399752] cx88[0]/2: subsystem: 18ac:db10, 
board: DViCO FusionHDTV DVB-T Plus [card=21]
Jan 19 19:24:32 phi kernel: [15167.399757] cx88[0]/2-dvb: cx8802_dvb_probe
Jan 19 19:24:32 phi kernel: [15167.399761] cx88[0]/2-dvb:  ->being probed by 
Card=21 Name=cx88[0], PCI 01:06
Jan 19 19:24:32 phi kernel: [15167.399766] cx88[0]/2: cx2388x based DVB/ATSC 
card
Jan 19 19:24:32 phi kernel: [15167.399771] cx8802_dvb_probe() failed to get 
frontend(1)
Jan 19 19:24:32 phi kernel: [15167.399775] cx88[0]/2: dvb_register failed (err 
= -22)
Jan 19 19:24:32 phi kernel: [15167.399779] cx88[0]/2: cx8802 probe failed, err 
= -22

rmmod cx8802:
Jan 19 19:25:06 phi kernel: [15200.890797] cx88-mpeg driver manager 
0000:01:06.2: PCI INT A disabled

modprobe cx88_dvb:
Jan 19 19:25:07 phi kernel: [15201.996410] cx88/2: cx2388x MPEG-TS Driver 
Manager version 0.0.6 loaded
Jan 19 19:25:07 phi kernel: [15201.996464] cx88[0]/2: cx2388x 8802 Driver 
Manager
Jan 19 19:25:07 phi kernel: [15201.996489] cx88-mpeg driver manager 
0000:01:06.2: PCI INT A -> Link[APC1] -> GSI 16 (level, low) -> IRQ 16
Jan 19 19:25:07 phi kernel: [15201.996502] cx88[0]/2: found at 0000:01:06.2, 
rev: 5, irq: 16, latency: 32, mmio: 0xfb000000
Jan 19 19:25:07 phi kernel: [15201.996516] cx8802_probe() allocating 1 
frontend(s)
Jan 19 19:25:07 phi kernel: [15202.001647] cx88/2: cx2388x dvb driver version 
0.0.6 loaded
Jan 19 19:25:07 phi kernel: [15202.001655] cx88/2: registering cx8802 driver, 
type: dvb access: shared
Jan 19 19:25:07 phi kernel: [15202.001661] cx88[0]/2: subsystem: 18ac:db10, 
board: DViCO FusionHDTV DVB-T Plus [card=21]
Jan 19 19:25:07 phi kernel: [15202.001667] cx88[0]/2-dvb: cx8802_dvb_probe
Jan 19 19:25:07 phi kernel: [15202.001671] cx88[0]/2-dvb:  ->being probed by 
Card=21 Name=cx88[0], PCI 01:06
Jan 19 19:25:07 phi kernel: [15202.001676] cx88[0]/2: cx2388x based DVB/ATSC 
card
Jan 19 19:25:07 phi kernel: [15202.002518] DVB: registering new adapter 
(cx88[0])
Jan 19 19:25:07 phi kernel: [15202.002526] DVB: registering adapter 0 frontend 
0 (Zarlink MT352 DVB-T)...

Note the "failed to get frontend(1)" in the first modprobe.

> Since modprobe after the crash failed with -ENODEV ["...cx88-dvb.ko): No
> such device], a quick grep through cx88-dvb.c in the source shows that
> that can only happen in a few places.  It should be easy to spot in the
> logs.  Adding an
>
> 	options cx88_dvb debug=1
>
> line to /etc/modprobe.conf would make things easier to see in the logs
> as well.

Thanks, I didn't know about that. The above syslog info ought to have been 
produced with that option in effect.

> Also what is the source of your cx88 driver: Debian Testing or the
> v4l-dvb repo?

It comes in the stock kernels that I built and tested (2.6.26 was the Debian 
pre-packaged kernel, the others were built from kernel.org source via make-
kpkg).

> > Or care?
>
> The issue is usually not one of caring, but one of having
> time, the specific hardware, and ability to reporduce the problem.

Of course. When your first two emails about a problem get no response, it 
starts looking like a plausible explanation, you know?

Peace,
Brendon


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Andy Walls]

On Mon, 2009-01-19 at 19:43 +1000, Brendon Higgins wrote:
> Andy Walls wrote (Monday 19 January 2009):
> > On Mon, 2009-01-19 at 13:37 +1000, Brendon Higgins wrote:
> > > > [snip]
> >
> > Well, you have the clues actually; they're in your log files.
> 
> Thanks Andy for prompting me to look at things again. I've discovered 
> something interesting about the way runvdr, a script which tries to keep vdr 
> alive (might be Debian custom, I'm not sure), interacts with the modules. When 
> vdr crashes, runvdr attempts to remove and re-load the dvb modules before 
> restarting vdr. Presumably this is a workaround for any modules left in a bad 
> state.
> 
> Interestingly though, the script doesn't take the module dependency tree into 
> account, only the first level branches, and so most of its rmmod commands fail 
> since the modules are in use by other modules. In my case it attempts to 
> remove videobuf_dvb, cx88_dvb, and dvb_core, in that order. Only rmmod 
> cx88_dvb succeeds.
> 
> Now the really interesting part: soon after that, runvdr tries to modprobe all 
> those modules back in again. Trying to modprobe videobuf_dvb and dvb_core does 
> nothing since they weren't unloaded. But trying to modprobe cx88_dvb fails: 
> "FATAL: Error inserting cx88_dvb 
> (/lib/modules/2.6.28/kernel/drivers/media/video/cx88/cx88-dvb.ko): No such 
> device"
> 
> Only after "rmmod cx8802" can cx88_dvb be loaded successfully.
> 
> Finally I've found a way to reproduce the bug on command! Hurrah!
> 
> Summary procedure, starting with a working dvb:
> 1) rmmod cx88_dvb
> 2) modprobe cx88_dvb
> Error: No such device.
> 3) rmmod cx8802
> 4) modprobe cx88_dvb
> Success (and cx8802 is pulled in automatically)
> 
> So it seems there might be some sort of module interdependency not being taken 
> care of. 

Yes.  Mauro did some work to decouple these modules in very recent
changes.  I did some follow-up changes to fix frontend allocations.  You
may want to try the latest v4l-dvb repository.


> I'll try to report the runvdr tree problem to the relevant place 
> later.
> 
> > At the time of the crash, what shows up in dmesg, /var/log/messages, and
> > any log that VDR creates?
> 
> Here's what's in /var/log/syslog when I perform the above procedure (with my 
> best-guess labels as to when I did what):
> 
> rmmod cx88_dvb:
> Jan 19 19:24:27 phi kernel: [15162.725955] cx88/2: unregistering cx8802 
> driver, type: dvb access: shared
> Jan 19 19:24:27 phi kernel: [15162.725967] cx88[0]/2: subsystem: 18ac:db10, 
> board: DViCO FusionHDTV DVB-T Plus [card=21]
> Jan 19 19:24:27 phi kernel: [15162.725972] cx88[0]/2-dvb: cx8802_dvb_remove
> 
> modprobe cx88_dvb:
> Jan 19 19:24:32 phi kernel: [15167.399736] cx88/2: cx2388x dvb driver version 
> 0.0.6 loaded
> Jan 19 19:24:32 phi kernel: [15167.399744] cx88/2: registering cx8802 driver, 
> type: dvb access: shared
> Jan 19 19:24:32 phi kernel: [15167.399752] cx88[0]/2: subsystem: 18ac:db10, 
> board: DViCO FusionHDTV DVB-T Plus [card=21]
> Jan 19 19:24:32 phi kernel: [15167.399757] cx88[0]/2-dvb: cx8802_dvb_probe
> Jan 19 19:24:32 phi kernel: [15167.399761] cx88[0]/2-dvb:  ->being probed by 
> Card=21 Name=cx88[0], PCI 01:06
> Jan 19 19:24:32 phi kernel: [15167.399766] cx88[0]/2: cx2388x based DVB/ATSC 
> card
> Jan 19 19:24:32 phi kernel: [15167.399771] cx8802_dvb_probe() failed to get 
> frontend(1)

This happens in one of the functions that was recently modified.  Trying
the latest v4l-dvb repo may give different results.  I don't have any
cx8802 hardware, so I can't test your procedure.

> Jan 19 19:24:32 phi kernel: [15167.399775] cx88[0]/2: dvb_register failed (err 
> = -22)
> Jan 19 19:24:32 phi kernel: [15167.399779] cx88[0]/2: cx8802 probe failed, err 
> = -22





> rmmod cx8802:
> Jan 19 19:25:06 phi kernel: [15200.890797] cx88-mpeg driver manager 
> 0000:01:06.2: PCI INT A disabled
> 
> modprobe cx88_dvb:
> Jan 19 19:25:07 phi kernel: [15201.996410] cx88/2: cx2388x MPEG-TS Driver 
> Manager version 0.0.6 loaded
> Jan 19 19:25:07 phi kernel: [15201.996464] cx88[0]/2: cx2388x 8802 Driver 
> Manager
> Jan 19 19:25:07 phi kernel: [15201.996489] cx88-mpeg driver manager 
> 0000:01:06.2: PCI INT A -> Link[APC1] -> GSI 16 (level, low) -> IRQ 16
> Jan 19 19:25:07 phi kernel: [15201.996502] cx88[0]/2: found at 0000:01:06.2, 
> rev: 5, irq: 16, latency: 32, mmio: 0xfb000000
> Jan 19 19:25:07 phi kernel: [15201.996516] cx8802_probe() allocating 1 
> frontend(s)
> Jan 19 19:25:07 phi kernel: [15202.001647] cx88/2: cx2388x dvb driver version 
> 0.0.6 loaded
> Jan 19 19:25:07 phi kernel: [15202.001655] cx88/2: registering cx8802 driver, 
> type: dvb access: shared
> Jan 19 19:25:07 phi kernel: [15202.001661] cx88[0]/2: subsystem: 18ac:db10, 
> board: DViCO FusionHDTV DVB-T Plus [card=21]
> Jan 19 19:25:07 phi kernel: [15202.001667] cx88[0]/2-dvb: cx8802_dvb_probe
> Jan 19 19:25:07 phi kernel: [15202.001671] cx88[0]/2-dvb:  ->being probed by 
> Card=21 Name=cx88[0], PCI 01:06
> Jan 19 19:25:07 phi kernel: [15202.001676] cx88[0]/2: cx2388x based DVB/ATSC 
> card
> Jan 19 19:25:07 phi kernel: [15202.002518] DVB: registering new adapter 
> (cx88[0])
> Jan 19 19:25:07 phi kernel: [15202.002526] DVB: registering adapter 0 frontend 
> 0 (Zarlink MT352 DVB-T)...
> 
> Note the "failed to get frontend(1)" in the first modprobe.
> 
> > Since modprobe after the crash failed with -ENODEV ["...cx88-dvb.ko): No
> > such device], a quick grep through cx88-dvb.c in the source shows that
> > that can only happen in a few places.  It should be easy to spot in the
> > logs.  Adding an
> >
> > 	options cx88_dvb debug=1
> >
> > line to /etc/modprobe.conf would make things easier to see in the logs
> > as well.
> 
> Thanks, I didn't know about that. The above syslog info ought to have been 
> produced with that option in effect.
> 
> > Also what is the source of your cx88 driver: Debian Testing or the
> > v4l-dvb repo?
> 
> It comes in the stock kernels that I built and tested (2.6.26 was the Debian 
> pre-packaged kernel, the others were built from kernel.org source via make-
> kpkg).
> 
> > > Or care?
> >
> > The issue is usually not one of caring, but one of having
> > time, the specific hardware, and ability to reporduce the problem.
> 
> Of course. When your first two emails about a problem get no response, it 
> starts looking like a plausible explanation, you know?

Yes.

Now that you have repeatable steps someone with hardware should be able
to fix the problem.  I don't have cx8802 hardware.

Regards,
Andy


> Peace,
> Brendon
> 

Re: Regression since 2.6.25 kernel: Crash of userspace program leaves DVB device unusable

[Brendon Higgins]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

Andy Walls wrote (Tuesday 20 January 2009):
> On Mon, 2009-01-19 at 19:43 +1000, Brendon Higgins wrote:
> > Summary procedure, starting with a working dvb:
> > 1) rmmod cx88_dvb
> > 2) modprobe cx88_dvb
> > Error: No such device.
> > 3) rmmod cx8802
> > 4) modprobe cx88_dvb
> > Success (and cx8802 is pulled in automatically)
> >
> > So it seems there might be some sort of module interdependency not being
> > taken care of.
>
> Yes.  Mauro did some work to decouple these modules in very recent
> changes.  I did some follow-up changes to fix frontend allocations.  You
> may want to try the latest v4l-dvb repository.

Just did that, grabbed v4l-dvb-2ed72b192848, and it seems to be fixed there. No 
complaints when rmmoding and modprobing the modules. There's something about 
it that xine and mplayer don't seem to like (though I can't be sure they 
worked previously - FWIW, xine won't change channels and when you try it 
causes "dvb_demux_feed_del: feed not in list (type=0 state=0 pid=ffff)" in 
syslog, mplayer... well I haven't figured out the interface, anyway), but vdr 
seems just fine with it, which is my primary concern.

It would be my luck that by the time I figure out what's actually going on I 
can't help because it's already fixed. :-)

Peace,
Brendon

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]