HD-PVR fails consistently on Linux, works on Windows

HD-PVR fails consistently on Linux, works on Windows

[Keith Pyle]

I recently purchased a Hauppauge HD-PVR (the 1212 version, label on 
bottom 49001LF, Rev F2).  I have consistent capture failures on Linux 
where data from the device simply stops, generally within a few minutes 
of starting a capture.  Yet, the device works flawlessly on Windows with 
the same USB and component cables, same power supply, and same physical 
position.  This suggests that the device itself has acceptable power, is 
not overheating, etc.  I'll detail below the testing I've done thus far 
and would appreciate any suggestions on how to further test or address 
the problem.

The good news is that I have a highly reproducible failure on Linux, but 
then that's the bad news too.

Thanks.

Keith

-- Linux tests --
I started trying to use the HD-PVR directly with my MythTV backend. I 
have subsequently switched all of my testing to simple direct captures 
from the /dev/video? device using /bin/cat to eliminate as many 
variables as possible.

I've done a large number of tests with combinations of the following:

OS: gentoo 3.4.7, gentoo 3.5.4
HD-PVR firmware: 1.5.7.0 (0x15), 1.7.1.30059 (0x1e)
Input resolution: fixed to 720p, fixed to 1080i, floating based on input
USB ports: motherboard ports on Intel DP45SG, motherboard ports on MSI 
X58 Pro-E, ports on SIIG USB PCIe card

Captures fail consistently.

I've verified that the HD-PVR is the only device on the USB bus and that 
the bus shows as "Linux Foundation 2.0 root hub" in all tests. I've 
increased the debug output level for the hdpvr driver to 6 
(/sys/module/hdpvr/parameters/hdpvr_debug) and collected the following:

Sep 21 17:01:00 mythbe kernel: [535043.504450] usb 9-1: New USB device 
found, idVendor=2040, idProduct=4903
Sep 21 17:01:00 mythbe kernel: [535043.504453] usb 9-1: New USB device 
strings: Mfr=1, Product=2, SerialNumber=3
Sep 21 17:01:00 mythbe kernel: [535043.504456] usb 9-1: Product: 
Hauppauge HD PVR
Sep 21 17:01:00 mythbe kernel: [535043.504458] usb 9-1: Manufacturer: AMBA
Sep 21 17:01:00 mythbe kernel: [535043.504459] usb 9-1: SerialNumber: 
00A6DD48
Sep 21 17:01:00 mythbe kernel: [535043.504523] usb 9-1: ep 0x1 - 
rounding interval to 32768 microframes, ep desc says 0 microframes
Sep 21 17:01:00 mythbe kernel: [535043.504528] usb 9-1: ep 0x81 - 
rounding interval to 32768 microframes, ep desc says 0 microframes
Sep 21 17:01:01 mythbe kernel: [535043.703947] hdpvr 9-1:1.0: firmware 
version 0x15 dated Jun 17 2010 09:26:53
Sep 21 17:01:01 mythbe kernel: [535043.889144] IR keymap rc-hauppauge 
not found
Sep 21 17:01:01 mythbe kernel: [535043.889146] Registered IR keymap 
rc-empty
Sep 21 17:01:01 mythbe kernel: [535043.889190] input: i2c IR (HD-PVR) as 
/devices/virtual/rc/rc5/input16
Sep 21 17:01:01 mythbe kernel: [535043.889415] rc5: i2c IR (HD-PVR) as 
/devices/virtual/rc/rc5
Sep 21 17:01:01 mythbe kernel: [535043.889417] ir-kbd-i2c: i2c IR 
(HD-PVR) detected at i2c-8/8-0071/ir0 [Hauppage HD PVR I2C]
Sep 21 17:01:01 mythbe kernel: [535043.889518] hdpvr 9-1:1.0: device now 
attached to video6
Sep 21 17:01:01 mythbe kernel: [535043.889534] usbcore: registered new 
interface driver hdpvr
Sep 21 17:05:11 mythbe kernel: [535293.776318] hdpvr 9-1:1.0: video 
signal: 1920x1080@30hz
Sep 21 17:05:14 mythbe kernel: [535297.312589] hdpvr 9-1:1.0: encoder 
start control request returned 0
Sep 21 17:05:15 mythbe kernel: [535297.670830] hdpvr 9-1:1.0: config 
call request for value 0x700 returned 1
Sep 21 17:05:15 mythbe kernel: [535297.670833] hdpvr 9-1:1.0: streaming 
started
Sep 21 17:05:15 mythbe kernel: [535297.670839] hdpvr 9-1:1.0: 
hdpvr_read:442 buffer stat: 64 free, 0 proc
Sep 21 17:05:15 mythbe kernel: [535297.670882] hdpvr 9-1:1.0: 
hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc
Sep 21 17:05:15 mythbe kernel: [535297.709079] hdpvr 9-1:1.0: 
hdpvr_read:502 buffer stat: 1 free, 63 proc
Sep 21 17:05:15 mythbe kernel: [535297.709088] hdpvr 9-1:1.0: 
hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc

(many repeats of the above two line sequence)

Sep 21 17:17:09 mythbe kernel: [536011.936858] hdpvr 9-1:1.0: 
hdpvr_read:502 buffer stat: 1 free, 63 proc
Sep 21 17:17:09 mythbe kernel: [536011.936866] hdpvr 9-1:1.0: 
hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc
Sep 21 17:17:36 mythbe kernel: [536038.853044] hdpvr 9-1:1.0: config 
call request for value 0x800 returned -110
Sep 21 17:17:36 mythbe kernel: [536038.853052] hdpvr 9-1:1.0: transmit 
worker exited
Sep 21 17:17:36 mythbe kernel: [536038.996035] hdpvr 9-1:1.0: used 0 
urbs to empty device buffers

If I understand correctly, this is showing a ETIMEDOUT error.  When I've 
looked at the cat with strace, it is always blocked on a read. So, it 
seems like the HD-PVR just stops sending.

I also ran a USB capture with wireshark and see much the same thing.  
While I haven't tried to decode the USB packets, the pattern is that the 
HD-PVR sends, the host sends a message/ack, this pattern repeats, and 
then nothing.  The majority of the failures occur in less than 15 minutes.

-- Windows tests --

I installed the Hauppauge software on a Windows 7 system and moved the 
USB cable to the Windows system.  Nothing else was changed. I've run 
many hours of successful captures.  I've checked each recording with 
ffprobe and verified that each is the expected length (i.e., no data 
drops at all).  The recordings are of good quality. This shows that the 
HD-PVR is capable of working as expected and quite reliably.

Re: HD-PVR fails consistently on Linux, works on Windows

[Jonathan]

On Wed, 26 Sep 2012 21:42:32 -0500
Keith Pyle <kpyle@austin.rr.com> wrote:

> I recently purchased a Hauppauge HD-PVR (the 1212 version, label on 
> bottom 49001LF, Rev F2).  I have consistent capture failures on Linux 
> where data from the device simply stops, generally within a few minutes 
> of starting a capture.  Yet, the device works flawlessly on Windows with 
> the same USB and component cables, same power supply, and same physical 
> position.  This suggests that the device itself has acceptable power, is 
> not overheating, etc.  I'll detail below the testing I've done thus far 
> and would appreciate any suggestions on how to further test or address 
> the problem.
> 
> The good news is that I have a highly reproducible failure on Linux, but 
> then that's the bad news too.
> 
> Thanks.
> 
> Keith
> 
> -- Linux tests --
> I started trying to use the HD-PVR directly with my MythTV backend. I 
> have subsequently switched all of my testing to simple direct captures 
> from the /dev/video? device using /bin/cat to eliminate as many 
> variables as possible.
> 
> I've done a large number of tests with combinations of the following:
> 
> OS: gentoo 3.4.7, gentoo 3.5.4
> HD-PVR firmware: 1.5.7.0 (0x15), 1.7.1.30059 (0x1e)
> Input resolution: fixed to 720p, fixed to 1080i, floating based on input
> USB ports: motherboard ports on Intel DP45SG, motherboard ports on MSI 
> X58 Pro-E, ports on SIIG USB PCIe card
> 
> Captures fail consistently.
> 
> I've verified that the HD-PVR is the only device on the USB bus and that 
> the bus shows as "Linux Foundation 2.0 root hub" in all tests. I've 
> increased the debug output level for the hdpvr driver to 6 
> (/sys/module/hdpvr/parameters/hdpvr_debug) and collected the following:
> 
> Sep 21 17:01:00 mythbe kernel: [535043.504450] usb 9-1: New USB device 
> found, idVendor=2040, idProduct=4903
> Sep 21 17:01:00 mythbe kernel: [535043.504453] usb 9-1: New USB device 
> strings: Mfr=1, Product=2, SerialNumber=3
> Sep 21 17:01:00 mythbe kernel: [535043.504456] usb 9-1: Product: 
> Hauppauge HD PVR
> Sep 21 17:01:00 mythbe kernel: [535043.504458] usb 9-1: Manufacturer: AMBA
> Sep 21 17:01:00 mythbe kernel: [535043.504459] usb 9-1: SerialNumber: 
> 00A6DD48
> Sep 21 17:01:00 mythbe kernel: [535043.504523] usb 9-1: ep 0x1 - 
> rounding interval to 32768 microframes, ep desc says 0 microframes
> Sep 21 17:01:00 mythbe kernel: [535043.504528] usb 9-1: ep 0x81 - 
> rounding interval to 32768 microframes, ep desc says 0 microframes
> Sep 21 17:01:01 mythbe kernel: [535043.703947] hdpvr 9-1:1.0: firmware 
> version 0x15 dated Jun 17 2010 09:26:53
> Sep 21 17:01:01 mythbe kernel: [535043.889144] IR keymap rc-hauppauge 
> not found
> Sep 21 17:01:01 mythbe kernel: [535043.889146] Registered IR keymap 
> rc-empty
> Sep 21 17:01:01 mythbe kernel: [535043.889190] input: i2c IR (HD-PVR) as 
> /devices/virtual/rc/rc5/input16
> Sep 21 17:01:01 mythbe kernel: [535043.889415] rc5: i2c IR (HD-PVR) as 
> /devices/virtual/rc/rc5
> Sep 21 17:01:01 mythbe kernel: [535043.889417] ir-kbd-i2c: i2c IR 
> (HD-PVR) detected at i2c-8/8-0071/ir0 [Hauppage HD PVR I2C]
> Sep 21 17:01:01 mythbe kernel: [535043.889518] hdpvr 9-1:1.0: device now 
> attached to video6
> Sep 21 17:01:01 mythbe kernel: [535043.889534] usbcore: registered new 
> interface driver hdpvr
> Sep 21 17:05:11 mythbe kernel: [535293.776318] hdpvr 9-1:1.0: video 
> signal: 1920x1080@30hz
> Sep 21 17:05:14 mythbe kernel: [535297.312589] hdpvr 9-1:1.0: encoder 
> start control request returned 0
> Sep 21 17:05:15 mythbe kernel: [535297.670830] hdpvr 9-1:1.0: config 
> call request for value 0x700 returned 1
> Sep 21 17:05:15 mythbe kernel: [535297.670833] hdpvr 9-1:1.0: streaming 
> started
> Sep 21 17:05:15 mythbe kernel: [535297.670839] hdpvr 9-1:1.0: 
> hdpvr_read:442 buffer stat: 64 free, 0 proc
> Sep 21 17:05:15 mythbe kernel: [535297.670882] hdpvr 9-1:1.0: 
> hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc
> Sep 21 17:05:15 mythbe kernel: [535297.709079] hdpvr 9-1:1.0: 
> hdpvr_read:502 buffer stat: 1 free, 63 proc
> Sep 21 17:05:15 mythbe kernel: [535297.709088] hdpvr 9-1:1.0: 
> hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc
> 
> (many repeats of the above two line sequence)
> 
> Sep 21 17:17:09 mythbe kernel: [536011.936858] hdpvr 9-1:1.0: 
> hdpvr_read:502 buffer stat: 1 free, 63 proc
> Sep 21 17:17:09 mythbe kernel: [536011.936866] hdpvr 9-1:1.0: 
> hdpvr_submit_buffers:209 buffer stat: 0 free, 64 proc
> Sep 21 17:17:36 mythbe kernel: [536038.853044] hdpvr 9-1:1.0: config 
> call request for value 0x800 returned -110
> Sep 21 17:17:36 mythbe kernel: [536038.853052] hdpvr 9-1:1.0: transmit 
> worker exited
> Sep 21 17:17:36 mythbe kernel: [536038.996035] hdpvr 9-1:1.0: used 0 
> urbs to empty device buffers
> 
> If I understand correctly, this is showing a ETIMEDOUT error.  When I've 
> looked at the cat with strace, it is always blocked on a read. So, it 
> seems like the HD-PVR just stops sending.
> 
> I also ran a USB capture with wireshark and see much the same thing.  
> While I haven't tried to decode the USB packets, the pattern is that the 
> HD-PVR sends, the host sends a message/ack, this pattern repeats, and 
> then nothing.  The majority of the failures occur in less than 15 minutes.
> 
> -- Windows tests --
> 
> I installed the Hauppauge software on a Windows 7 system and moved the 
> USB cable to the Windows system.  Nothing else was changed. I've run 
> many hours of successful captures.  I've checked each recording with 
> ffprobe and verified that each is the expected length (i.e., no data 
> drops at all).  The recordings are of good quality. This shows that the 
> HD-PVR is capable of working as expected and quite reliably.

It may be a coincidence  but I since I started using  irqbalance ( https://code.google.com/p/irqbalance/ ) my HD-PVR has been  completely stable.  Before that I was experiencing daily lockups. 

Re: HD-PVR fails consistently on Linux, works on Windows

[David Röthlisberger]

On Wed, 26 Sep 2012 21:42:32 -0500
Keith Pyle <kpyle@austin.rr.com> wrote:
> I recently purchased a Hauppauge HD-PVR (the 1212 version, label on 
> bottom 49001LF, Rev F2).  I have consistent capture failures on Linux 
> where data from the device simply stops, generally within a few minutes 
> of starting a capture.
> 
> [...]
> 
> Sep 21 17:01:01 mythbe kernel: [535043.703947] hdpvr 9-1:1.0: firmware 
> version 0x15 dated Jun 17 2010 09:26:53

When we contacted Hauppauge regarding the stability issue, they
recommended upgrading to the latest firmware dated Mar 26 2012.
We *think* this has improved stability, but it certainly hasn't
fixed it completely.

Upgrading the firmware requires a Windows PC -- see
http://www.hauppauge.com/site/support/support_hdpvr.html


On 13 Oct 2012, at 16:28, Jonathan wrote:

> It may be a coincidence but I since I started using irqbalance (
> https://code.google.com/p/irqbalance/ ) my HD-PVR has been completely
> stable. Before that I was experiencing daily lockups.

Interesting. You definitely didn't upgrade the firmware around the same
time?

We think the stability is worse when the Linux PC is heavily loaded: We
do real-time image processing on the video stream from the HD PVR, so
the CPUs are maxed out, and we get frequent lock-ups. We also think the
lock-ups are more frequent when we have several HD PVRs connected to the
same PC, all running at the same time. I'll have to try this irqbalance.

--Dave.

Re: HD-PVR fails consistently on Linux, works on Windows

[Keith Pyle]

On 10/13/12 14:17, David Röthlisberger wrote:
> On Wed, 26 Sep 2012 21:42:32 -0500
> Keith Pyle <kpyle@austin.rr.com> wrote:
>> I recently purchased a Hauppauge HD-PVR (the 1212 version, label on
>> bottom 49001LF, Rev F2).  I have consistent capture failures on Linux
>> where data from the device simply stops, generally within a few minutes
>> of starting a capture.
>>
>> [...]
>>
>> Sep 21 17:01:01 mythbe kernel: [535043.703947] hdpvr 9-1:1.0: firmware
>> version 0x15 dated Jun 17 2010 09:26:53
> When we contacted Hauppauge regarding the stability issue, they
> recommended upgrading to the latest firmware dated Mar 26 2012.
> We *think* this has improved stability, but it certainly hasn't
> fixed it completely.
>
> Upgrading the firmware requires a Windows PC -- see
> http://www.hauppauge.com/site/support/support_hdpvr.html
The Mar 26, 2012 firmware is the 1.7.1.30059 version that shows as 0x1e 
in the kernel messages.  In my case, I've tried both 0x1e and 0x15 with 
no discernible difference in stability.  That is, both hang reliably and 
repeatedly on Linux but not when using a Windows system.
>
> On 13 Oct 2012, at 16:28, Jonathan wrote:
>
>> It may be a coincidence but I since I started using irqbalance (
>> https://code.google.com/p/irqbalance/ ) my HD-PVR has been completely
>> stable. Before that I was experiencing daily lockups.
> Interesting. You definitely didn't upgrade the firmware around the same
> time?
>
> We think the stability is worse when the Linux PC is heavily loaded: We
> do real-time image processing on the video stream from the HD PVR, so
> the CPUs are maxed out, and we get frequent lock-ups. We also think the
> lock-ups are more frequent when we have several HD PVRs connected to the
> same PC, all running at the same time. I'll have to try this irqbalance.
>
> --Dave.
>
All of my tests have been using only one HD-PVR on otherwise nearly idle 
systems with *lots* of system resources available.  One test system is 
an Intel Core 2 Quad Q9400 (2.66 GHz) and the other is an Intel i7 950 
(3.0 GHz).  CPU loads are low during captures - not anywhere near having 
any core at 100%.  While system load could certainly be a factor for 
some, I do not believe it is in my tests.

Keith

Re: HD-PVR fails consistently on Linux, works on Windows

[Jonathan]

On Sat, 13 Oct 2012 20:17:59 +0100
David Röthlisberger <david@rothlis.net> wrote:

> On Wed, 26 Sep 2012 21:42:32 -0500
> Keith Pyle <kpyle@austin.rr.com> wrote:
> > I recently purchased a Hauppauge HD-PVR (the 1212 version, label on 
> > bottom 49001LF, Rev F2).  I have consistent capture failures on Linux 
> > where data from the device simply stops, generally within a few minutes 
> > of starting a capture.
> > 
> > [...]
> > 
> > Sep 21 17:01:01 mythbe kernel: [535043.703947] hdpvr 9-1:1.0: firmware 
> > version 0x15 dated Jun 17 2010 09:26:53
> 
> When we contacted Hauppauge regarding the stability issue, they
> recommended upgrading to the latest firmware dated Mar 26 2012.
> We *think* this has improved stability, but it certainly hasn't
> fixed it completely.
> 
> Upgrading the firmware requires a Windows PC -- see
> http://www.hauppauge.com/site/support/support_hdpvr.html
> 
> 
> On 13 Oct 2012, at 16:28, Jonathan wrote:
> 
> > It may be a coincidence but I since I started using irqbalance (
> > https://code.google.com/p/irqbalance/ ) my HD-PVR has been completely
> > stable. Before that I was experiencing daily lockups.
> 
> Interesting. You definitely didn't upgrade the firmware around the same
> time?
> 
> We think the stability is worse when the Linux PC is heavily loaded: We
> do real-time image processing on the video stream from the HD PVR, so
> the CPUs are maxed out, and we get frequent lock-ups. We also think the
> lock-ups are more frequent when we have several HD PVRs connected to the
> same PC, all running at the same time. I'll have to try this irqbalance.
> 
> --Dave.

No change in my firmware; still on version 0x15. FWIW, after using irqblance for about 10 days, cat /proc/interrupts shows the interrupt for xhci_hcd (the USB3 bus my HD-PVR is attached to) is now spread across all 4 cores whereas before it was loaded up on CPU0.  Same thing was shown for  ahci,  and a lot of data is being written to disk when the HD-PVR is working so I guess that could be a factor as well.

Jonathan

Re: HD-PVR fails consistently on Linux, works on Windows

[Keith Pyle]

On 09/26/12 21:42, Keith Pyle wrote:
> I recently purchased a Hauppauge HD-PVR (the 1212 version, label on 
> bottom 49001LF, Rev F2).  I have consistent capture failures on Linux 
> where data from the device simply stops, generally within a few 
> minutes of starting a capture.  Yet, the device works flawlessly on 
> Windows with the same USB and component cables, same power supply, and 
> same physical position.  This suggests that the device itself has 
> acceptable power, is not overheating, etc.  I'll detail below the 
> testing I've done thus far and would appreciate any suggestions on how 
> to further test or address the problem.
>
> The good news is that I have a highly reproducible failure on Linux, 
> but then that's the bad news too.
>
> Thanks.
>
> Keith
>
> -- Linux tests --
> I started trying to use the HD-PVR directly with my MythTV backend. I 
> have subsequently switched all of my testing to simple direct captures 
> from the /dev/video? device using /bin/cat to eliminate as many 
> variables as possible.
>
> I've done a large number of tests with combinations of the following:
>
> OS: gentoo 3.4.7, gentoo 3.5.4
> HD-PVR firmware: 1.5.7.0 (0x15), 1.7.1.30059 (0x1e)
> Input resolution: fixed to 720p, fixed to 1080i, floating based on input
> USB ports: motherboard ports on Intel DP45SG, motherboard ports on MSI 
> X58 Pro-E, ports on SIIG USB PCIe card
>
> Captures fail consistently.
Here's some interesting new information.  On noticing that the 3.6 
kernel included several USB related commits, I updated the kernel of my 
test system (MSI X58 Pro-E) from 3.5.4 to 3.6.2.  I ran a series of 
capture tests with firmware 0x1e and all other variables exactly as 
before - same USB cable, port, physical position, etc.  I have 20 
successful 1-hour captures and 1 failure.  There are no logged messages 
for the failed capture but the timing coincides within a minute after 
initializing a wireless joystick on a different USB bus.  I've been 
unable to reproduce this failure, so I cannot conclusively state that 
there is a correlation.

The results of the kernel change are dramatic.  Under 3.5.x and earlier, 
the failure rate for 1-hour captures was 100%.  Most failed in less than 
10 minutes.  There were some instances of hard hangs on the HD-PVR 
(i.e., power cycle required).  With 3.6.2, it is less than 5% failure.  
The one failure was a truncated recording.

I am continuing capture tests on my test system to build more data. I 
will next be updating my MythTV backend to 3.6 and trying the capture 
tests on it since it has a different motherboard.

It would be quite interesting if others with HD-PVR problems see similar 
results for 3.6.2 or better.

Keith

Re: HD-PVR fails consistently on Linux, works on Windows

[David Röthlisberger]

On 17 Oct 2012, at 18:54, Keith Pyle wrote:
> On 09/26/12 21:42, Keith Pyle wrote:
> > I recently purchased a Hauppauge HD-PVR (the 1212 version, label on bottom
> > 49001LF, Rev F2). I have consistent capture failures on Linux where data from
> > the device simply stops, generally within a few minutes of starting a capture.
> > Yet, the device works flawlessly on Windows with the same USB and component
> > cables, same power supply, and same physical position. This suggests that the
> > device itself has acceptable power, is not overheating, etc. I'll detail below
> > the testing I've done thus far and would appreciate any suggestions on how to
> > further test or address the problem.
> 
> Here's some interesting new information. On noticing that the 3.6 kernel
> included several USB related commits, I updated the kernel of my test system
> (MSI X58 Pro-E) from 3.5.4 to 3.6.2. I ran a series of capture tests with
> firmware 0x1e and all other variables exactly as before - same USB cable, port,
> physical position, etc. I have 20 successful 1-hour captures and 1 failure.

We have run extensive tests comparing the HD PVR's stability on kernel 3.6.6
vs. kernel <3.6; having a single HD PVR connected to the PC vs. multiple HD
PVRs on the same USB controller; adding better heatsinks; running irqbalance.
None of these made any difference. The only significant improvement we found
was from removing the lid and adding fans pointing directly at the circuit
board. Even this didn't cure the problem completely, it only increased the
time-to-failure.

For details, including the time-to-failure measurements and the shell script we
used, see http://stb-tester.com/hardware.html#hauppauge-hd-pvr

Note that our workload is quite different from yours: An individual recording
typically lasts 2-10 minutes, but as soon as it has finished we start another
one, so we are stopping and starting all day. Hauppauge have hinted that under
these conditions the HD PVR is unreliable on Windows too, which implies a
problem in the hardware or firmware, rather than the Linux hdpvr driver.

Dave.