Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

Hi Keir,

I recieved this reply to my request for help with ivtv on Xen on the
ivtv-devel list. I don't feel very well qualified to respond to
Andrew. Can I ask what your thoughts are? Am I just out of luck?

Thanks,
Dave

On 7/7/05, Andrew May <acmay@acmay.homeip.net> wrote:
> On Wed, 2005-07-06 at 10:20 -0400, David Muench wrote:
> > "I think the ivtv driver is probably not calculating dma addresses in
> > the way that xen requires. On native Linux, if you allocate a
> > multi-page chunk of physical memory, you can pass the start address of
> > that buffer to hardware and it can dma the entire buffer given just
> > that address. In Xen, because we give guests 'pseudo-physical' memory,
> > that physical buffer may not be really physically contiguous. So we
> > need drivers to dma_alloc_coherent or pci_alloc_consistent the memory
> > they will use for dma --- we modified those functions to ensure they
> > return suitable contiguous physical memory."
> 
> The Xen people seem to be a bit off here. You may want to look at
> linux/Documentation/DMA-mapping.txt.
> 
> But alloc_coherent and alloc_consistent are for uncached mem for DMA and
> that is very different than normal mem for DMA. That could really slow
> things down when getting the data out of buffers latter.
> 
> pci_map_single is correct for doing bulk dma to normal kmalloc data.
> 
> The coherent/consistent mem doesn't make much sense on x86 stuff, but
> for other archs like some PPC ones without cache snooping mem it becomes
> a big issue. The CPU and the DMA device cannot write to the same
> cacheline of data without causing havoc. The CPU would work with
> the data in cache, but the DMA would go directly to DRAM. When the
> cache flush does happen it would overwrite half the work done.
> 
> This only comes into play on a smaller ring of descriptors for most HW.
> 
> If the bulk of the data is only touched by one device at a time, then
> the map_single function just needs to do a cache flush before the
> transfer happens along with the other work.
> 
> I am much more familar with network drivers where it is common for one
> consistent set of mem for the descriptors, but all the packets do just
> the map_single(). But since jumbo frames aren't typical yet, the packets
> all fit in one page.
> 
> So it looks like the Xen people have some work to do if they really want
> to support any type of DMA device.
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ivtv-devel
> 


-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 7 Jul 2005, at 17:07, David Muench wrote:

> I recieved this reply to my request for help with ivtv on Xen on the
> ivtv-devel list. I don't feel very well qualified to respond to
> Andrew. Can I ask what your thoughts are? Am I just out of luck?

Andrew's simply pointing out that the ivtv driver already uses the 
dma-mapping interface properly. It's good to have that confirmed. :-)

However, looking back at XenLinux's implementation of 
{dma,pci}_map_single, it is broken for multi-page buffers. It's 
surprising noone has hit a problem before now, but most devices other 
than video-capture cards you deal with only up to a page at a time.

So this is a Xen bug: I'll find some time to fix this and get back to 
you. Are you using Xen 2.0 or 3.0 (unstable)?

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/7/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 7 Jul 2005, at 17:07, David Muench wrote:
> 
> > I recieved this reply to my request for help with ivtv on Xen on the
> > ivtv-devel list. I don't feel very well qualified to respond to
> > Andrew. Can I ask what your thoughts are? Am I just out of luck?
> 
> Andrew's simply pointing out that the ivtv driver already uses the
> dma-mapping interface properly. It's good to have that confirmed. :-)
> 
> However, looking back at XenLinux's implementation of
> {dma,pci}_map_single, it is broken for multi-page buffers. It's
> surprising noone has hit a problem before now, but most devices other
> than video-capture cards you deal with only up to a page at a time.
> 
> So this is a Xen bug: I'll find some time to fix this and get back to
> you. Are you using Xen 2.0 or 3.0 (unstable)?

That's great news, I thought I was up a creek. I'm using 2.0.6. Thanks
again, I really appreciate your help.

Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Andrew May]

On Thu, 2005-07-07 at 13:34 -0400, David Muench wrote:
> On 7/7/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> > 
> > On 7 Jul 2005, at 17:07, David Muench wrote:
> > 
> > > I recieved this reply to my request for help with ivtv on Xen on the
> > > ivtv-devel list. I don't feel very well qualified to respond to
> > > Andrew. Can I ask what your thoughts are? Am I just out of luck?
> > 
> > Andrew's simply pointing out that the ivtv driver already uses the
> > dma-mapping interface properly. It's good to have that confirmed. :-)

I am not sure I can really confirm that, it was more the suggestion of
pci_alloc_consistent that looked very wrong. I only did a quick check in
ivtv-queue.c and most of the stuff looks in order. But since I don't
really know the real flow of data it is hard to say for sure. But given
that the driver works for me it must be correct. :>

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 7 Jul 2005, at 18:34, David Muench wrote:

>> So this is a Xen bug: I'll find some time to fix this and get back to
>> you. Are you using Xen 2.0 or 3.0 (unstable)?
>
> That's great news, I thought I was up a creek. I'm using 2.0.6. Thanks
> again, I really appreciate your help.

I've pushed the fix to our xen-2.0-testing hg repository on 
xenbits.xensource.com. If you don't want to use hg, I've also put a 
source tarball at http://www.cl.cam.ac.uk/~kaf24/xen-2.0-testing.tar.gz 
(you'll still need to build it yourself though).

Please let me know if this fixes your problem -- if so then I'll also 
apply the fix to our unstable repository.

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/8/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> I've pushed the fix to our xen-2.0-testing hg repository on
> xenbits.xensource.com. If you don't want to use hg, I've also put a
> source tarball at http://www.cl.cam.ac.uk/~kaf24/xen-2.0-testing.tar.gz
> (you'll still need to build it yourself though).
> 
> Please let me know if this fixes your problem -- if so then I'll also
> apply the fix to our unstable repository.

Hi Keir,

Looks like it didn't quite work. I started a capture and then ended it
20 seconds later, and I still have a zero byte file. Ivtv has some new
messages in dmesg though:

saa7115: decoder disable output
saa7115: decoder set input (8)
saa7115: now setting S-Video input
ivtv: Setting audio matrix to input 1, output 1
saa7115: decoder enable output
saa7115: decoder disable output
saa7115: decoder enable output
ivtv: DEC: REG_ENCSG1LEN wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
ivtv: ENC: REG_DMAXFER wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
ivtv: ENC: User stopped capture.
ivtv: ENC: DMA still Pending while stoping capture!
ivtv: ENC: REG_DMASTATUS2 wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000

Thanks,
Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 8 Jul 2005, at 16:38, David Muench wrote:

> Hi Keir,
>
> Looks like it didn't quite work. I started a capture and then ended it
> 20 seconds later, and I still have a zero byte file. Ivtv has some new
> messages in dmesg though:

Well, it's not clear whether this is a bug in Xen or the ivtv driver. 
:-)

Basically, my fix for dma_map_single() introduces bounce buffers. These 
should work okay if:
  (a) I haven't screwed up the bounce-buffer code; and
  (b) The ivtv driver is scrupulously careful in its use of the 
streaming DMA interface.

For example, it is not good for the driver to read/write the buffer 
without calling unmap, or sync_for_cpu. I see that the driver does do 
the right thing at least most of the time but, of course, if there are 
any bugs they won't actually cause problems on most architectures (i386 
in particular).

OTOH, the bounce buffer code is not exactly thoroughly tested, so it is 
feasible the bug lies there. Although the implementation is pretty 
simple.

I think the bottom line is that this will need fixing by someone with 
the hardware -- probably by adding tracing to the map/sync functions to 
check that they are used correctly, and to sanity-check the bounce 
buffer code.

Some more detail on how the bounce buffers are implemented. It's very 
similar to what arch/arm does for these streaming DMA calls:
  * dma_map_single() allocates a physically contiguous bounce buffer 
and, if dir!=DMA_FROM_DEVICE, copies the host buffer into the bounce 
buffer.
  * dma_unmap_single() copies bounce -> host (if dir!=DMA_TO_DEVICE) and 
then frees the bounce buffer
  * dma_sync_single_to_{cpu,device}() copy to/from the bounce buffer as 
appropriate, if teh direction argument is in the matching direction.

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 8 Jul 2005, at 16:38, David Muench wrote:

> Looks like it didn't quite work. I started a capture and then ended it
> 20 seconds later, and I still have a zero byte file. Ivtv has some new
> messages in dmesg though:

One last thing you might want to try is to edit 
linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that 
there are four separate uses of memcpy(), each conditional on a test of 
the form 'if (direction != DMA_xxx)'. You *might* have more success if 
you remove the direction-conditional tests and always memcpy.

Apart from that, this needs someone with hardware and expertise, and 
who cares about both ivtv and xen, to wade in. :-)

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/11/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> One last thing you might want to try is to edit
> linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that
> there are four separate uses of memcpy(), each conditional on a test of
> the form 'if (direction != DMA_xxx)'. You *might* have more success if
> you remove the direction-conditional tests and always memcpy.
> 
> Apart from that, this needs someone with hardware and expertise, and
> who cares about both ivtv and xen, to wade in. :-)

I'll give that a shot tonight, thanks for the suggestion. I really
appreciate your efforts and if that doesn't help, I'll see if anyone
on the ivtv-devel list is into Xen too. I wish I had the kernel
programming expertise myself. But at least the drivers work well in
dom0.

Stupid question, but is there any way to tell Xen to allocate a domU
contiguously? Or is that just not the Xen way, so to speak?

Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 11 Jul 2005, at 14:08, David Muench wrote:

> I'll give that a shot tonight, thanks for the suggestion. I really
> appreciate your efforts and if that doesn't help, I'll see if anyone
> on the ivtv-devel list is into Xen too. I wish I had the kernel
> programming expertise myself. But at least the drivers work well in
> dom0.
>
> Stupid question, but is there any way to tell Xen to allocate a domU
> contiguously? Or is that just not the Xen way, so to speak?

It would be tricky. Even if we did support it, if the domU is connected 
to a network interface, or you use the balloon driver, then that will 
cause memory to start getting fragmented. But maybe it would be good 
enough -- this fragmenting problem will also occur in domain0 over 
time.

I think you'll have to stick to dom0 for now and hope someone on the 
ivtv list has some spare cycles to help you out.

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/11/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> One last thing you might want to try is to edit
> linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that
> there are four separate uses of memcpy(), each conditional on a test of
> the form 'if (direction != DMA_xxx)'. You *might* have more success if
> you remove the direction-conditional tests and always memcpy.
> 
> Apart from that, this needs someone with hardware and expertise, and
> who cares about both ivtv and xen, to wade in. :-)

Well, life got in the way and I didn't try this fix until just now..
But it works! I'm going to get my mythtv setup moved into the domU so
I can test it completely, but it seems to work perfectly.

Thank you very, very much. Is removing the conditionals a permanent
solution that can be commited, or is there anything else you'd like me
to test?

Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 13 Jul 2005, at 20:51, David Muench wrote:

> Well, life got in the way and I didn't try this fix until just now..
> But it works! I'm going to get my mythtv setup moved into the domU so
> I can test it completely, but it seems to work perfectly.
>
> Thank you very, very much. Is removing the conditionals a permanent
> solution that can be commited, or is there anything else you'd like me
> to test?

I believe the code as checked in is strictly correct, but maybe we can 
relax things a little. What happens if you leave the direction tests in 
dma_map_single and dma_unmap_single, but remove them from 
dma_sync_single_for_cpu and dma_sync_single_for_device?

If the above works, leave the direction tests in the new modified form; 
otherwise revert to the previous working state (ie. direction tests 
removed from all four functions). Either way, another thing that is 
interesting to try is the following:
(1) At the start of dma_sync_single_for_cpu add:
  if (direction == DMA_TO_DEVICE)
     dma_sync_single_for_device(dev, dma_handle, size, direction);
(2) At the start of dma_sync_single_for_device add:
  if (direction == DMA_FROM_DEVICE)
     dma_sync_single_for_cpu(dev, dma_handle, size, direction);

This will make the sync functions behave as they do in the ARM 
architecture, which also uses bounce buffers but has at least been 
looked over by the core kernel DMA guys.

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/13/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> I believe the code as checked in is strictly correct, but maybe we can
> relax things a little. What happens if you leave the direction tests in
> dma_map_single and dma_unmap_single, but remove them from
> dma_sync_single_for_cpu and dma_sync_single_for_device?

That works fine.

> If the above works, leave the direction tests in the new modified form;
> otherwise revert to the previous working state (ie. direction tests
> removed from all four functions). Either way, another thing that is
> interesting to try is the following:
> (1) At the start of dma_sync_single_for_cpu add:
>   if (direction == DMA_TO_DEVICE)
>      dma_sync_single_for_device(dev, dma_handle, size, direction);
> (2) At the start of dma_sync_single_for_device add:
>   if (direction == DMA_FROM_DEVICE)
>      dma_sync_single_for_cpu(dev, dma_handle, size, direction);
> 
> This will make the sync functions behave as they do in the ARM
> architecture, which also uses bounce buffers but has at least been
> looked over by the core kernel DMA guys.

That doesn't work however:

saa7115: decoder disable output
saa7115: decoder enable output
ivtv: DEC: REG_ENCSG1LEN wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
ivtv: ENC: REG_DMAXFER wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
ivtv: ENC: User stopped capture.
ivtv: ENC: REG_DMASTATUS2 wait failed
ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001
dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003
ivtv: DMA DEC Buffers:
 0x00000000:0x00000000:0x00000000
ivtv: DMA ENC Buffers:
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000
 0x00000000:0x00000000:0x00000000

I'm going to take that out and run with the conditionals in the map
functions only, and see if that holds up under longer testing.

Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[Keir Fraser]

On 15 Jul 2005, at 13:43, David Muench wrote:

>> This will make the sync functions behave as they do in the ARM
>> architecture, which also uses bounce buffers but has at least been
>> looked over by the core kernel DMA guys.
>
> That doesn't work however:

Well, I think there is probably a grey area over what accesses the 
kernel can make to memory that it currently has mapped by 
dma_map_single.... I think I will just remove the direction checks from 
the sync functions, leave them in the map/unmap functions, and hope 
that suffices to get most drivers working.

Thanks for your testing efforts on this!

  -- Keir

Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?

[David Muench]

On 7/15/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> Well, I think there is probably a grey area over what accesses the
> kernel can make to memory that it currently has mapped by
> dma_map_single.... I think I will just remove the direction checks from
> the sync functions, leave them in the map/unmap functions, and hope
> that suffices to get most drivers working.
> 
> Thanks for your testing efforts on this!

No problem, thanks very much for all your help. It still amazes me
that a Celeron in my basement can have 2 domains running web servers,
1 doing dns/dhcp/tftp type services, a mysql domain, a domain running
Asterisk to control all my VOIP phones, and a mythtv backend domain
that can simultaneously record 3 DVD quality mpeg-2 streams off of
directv while playing back to (at least) two frontend machines in my
house. All thanks to Xen. And I still have another 200 meg of
unallocated ram, I wonder what I can do with that.. :)

Dave

-- 
David Muench - davemuench@gmail.com
Jabber ID: dave@jabber.wasteland.org