IVTV and xen: silent intermittent failures reading from /dev/video0

IVTV and xen: silent intermittent failures reading from /dev/video0

[Dan Schmick]

[-- Attachment #1.1: Type: text/plain, Size: 3178 bytes --]

[Disclaimer: I realize that I probably won't get this problem solved as it's probably an unsual setup. I thought I'd ask, though.] 

P roblem: 'cat /dev/video0 > test.mpg' (a simple IVTV test of my hardware mpeg2 encoder) doesn't work properly under Xen. When this command is run, test.mpg should grow approximately 1MB/sec in perpetuity, and does so when the machine is booted under a 'normal' (i.e. non-Xen) kernel. When I boot under Xen, test.mpg grows properly for a while (say, 2 to 10 minutes) and then stops growing. Typical final file sizes are 50MB to 500MB. Note that there are *no* errors in dmesg, syslog, etc. The cat command doesn't exit, either. It seems to be blocked on read. This same behavior is observed whether IVTV is in dom0 or domu. 

More information: 

- I can usually hasten the failure by doing something with the Xen machine that causes lots of disk and/or network activity. Failure-hastening activity can occur in any Xen domain, not just the one with /dev/video0 or Domain 0. 
- When 'cat /dev/video0' is running and IVTV is working, 'xm top' shows 3-4% cpu usage in the IVTV domu. Inside the IVTV domu, however, I see 1% cpu usage or so. As soon as the failure occurs, 'xm top' shows that the IVTV domu cpu usage has dropped to 1%. 

This failure occurs identically whether IVTV is in Domain 0 or in an unprivileged domain. No difference. 

My setup: 

- Hardware: Nforce2 motherboard, 2GB ram, 1.8GHz Athlon XP, Hauppauge PVR-150 
- Xen: 3.0.2 
- Kernel: 2.6.16-xen 
- ivtv: 0.6.6 

What I've tried so far: 

1. Verified that this is not a hardware problem (I've done stability testing, prime95, memtest86, etc. The system works perfectly and the IVTV device is rock-solid on a non-Xen Linux 2.6.16 kernel.) 
2. Moved the encoder (IVTV) to its own IRQ 
3. Set PCI latency timers to reasonable values for all devices (e.g. 128) 
4. Verified that there are *no* errors in dmesg, xm dmesg, syslogs on domu, syslogs on dom0, etc. 
5. Verified that there are no throughput problems with disks. The output of 'cat' is redirected to a dedicated disk on a dedicated controller where I can get 30MB/sec writing speeds sustained. 
6. Verified that I'm using the proper firmware for my encoder card 
7. Verified that swiotlb=force is not the problem. Using swiotlb=force is required to get IVTV working under Xen kernels but not under 'normal' kernels. However, passing it under a 'normal' 2.6.16 kernel doesn't cause any problems -- the IVTV device works fine. 
8. Tuned the Xen scheduler to reduce latency for Domain 0 and the domain with /dev/video0. This helped a little, but not much. 

Consequences of problem: 

My recordings 'hiccup' every 2-10 minutes or so. The hiccup is a jump in the recording of probably just a few seconds. I believe that mythtv detects that reading from the device has failed and restarts the read. These interruptions are annoying but I don't have a machine to dedicate to this purpose, so I'm stuck with them. If there's anything that I can do to help diagnose this problem please let me know. 

Xen rocks, by the way. Thanks very much for your work on this excellent platform, and for your time reading this post. 

Dan 

[-- Attachment #1.2: Type: text/html, Size: 3690 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: IVTV and xen: silent intermittent failures reading from /dev/video0

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 666 bytes --]

On 20/1/07 3:30 am, "Dan Schmick" <dschmick@dschmick.com> wrote:

> My recordings 'hiccup' every 2-10 minutes or so.  The hiccup is a jump in the
> recording of probably just a few seconds.  I believe that mythtv detects that
> reading from the device has failed and restarts the read. These interruptions
> are annoying but I don't have a machine to dedicate to this purpose, so I'm
> stuck with them.  If there's anything that I can do to help diagnose this
> problem please let me know.

What happens if you pin the ivtv domain to a CPU that no other domains run
on? E.g., ivtv on cpu1 and all others on cpu0? Perhaps preemption latency is
an issue. 

 -- Keir
 

[-- Attachment #1.2: Type: text/html, Size: 1159 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: IVTV and xen: silent intermittent failuresreading from /dev/video0

[Ian Pratt]

> 	My recordings 'hiccup' every 2-10 minutes or so.  The hiccup is
a
> jump in the recording of probably just a few seconds.  I believe that
> mythtv detects that reading from the device has failed and restarts
the
> read. These interruptions are annoying but I don't have a machine to
> dedicate to this purpose, so I'm stuck with them.  If there's anything
that
> I can do to help diagnose this problem please let me know.

There's been lots of scheduler improvements since 3.0.2 -- I'd suggest
upgrading first.

Ian
 

Re: IVTV and xen: silent intermittent failures reading from /dev/video0 (SOLVED)

[Dan Schmick]

Ian,

Upgrading to 3.0.3 didn't solve the problem, but upgrading to 3.0.4 did (so far, so good, anyway).  

I've been beating my head against this for a week.  Many thanks!

Dan

----- Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:
> > 	My recordings 'hiccup' every 2-10 minutes or so.  The hiccup is
> a
> > jump in the recording of probably just a few seconds.  I believe
> that
> > mythtv detects that reading from the device has failed and restarts
> the
> > read. These interruptions are annoying but I don't have a machine
> to
> > dedicate to this purpose, so I'm stuck with them.  If there's
> anything
> that
> > I can do to help diagnose this problem please let me know.
> 
> There's been lots of scheduler improvements since 3.0.2 -- I'd
> suggest
> upgrading first.
> 
> Ian
>  
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel