PCTV 290e page allocation failure

PCTV 290e page allocation failure

[Andy Furniss]

Hi.

I've got an old PIII 256M Ram PC that for years has served as an always 
on TV streamer/recorder and gateway/server box.

It had 2 PCI dvb cards until now I have a PCTV 290e so replaced one card 
with a USB2 PCI card.

At first the PCTV worked OK, but now I've been getting a page allocation 
failure. After this happens I can't use the PVTV until I re-plug it. It 
will then work, but the next day I am likely to get the same.

Free/meminfo (to my untrained eye) seem to show available mem.

mumudvb: page allocation failure: order:4, mode:0x80d0
Pid: 15776, comm: mumudvb Not tainted 3.3.0-rc1 #6
Call Trace:
  [<c102a43b>] ? printk+0x1b/0x20
  [<c107c353>] warn_alloc_failed+0xc3/0x130
  [<c107dab7>] __alloc_pages_nodemask+0x447/0x580
  [<c1006e58>] dma_generic_alloc_coherent+0x68/0xe0
  [<c1255b41>] hcd_buffer_alloc+0x101/0x110
  [<c1006df0>] ? dma_set_mask+0x50/0x50
  [<c1248e35>] usb_alloc_coherent+0x25/0x30
  [<d1dab1cc>] em28xx_init_isoc+0x10c/0x360 [em28xx]
  [<d1dcb0ad>] em28xx_start_feed+0xad/0xe0 [em28xx_dvb]
  [<d1dcca20>] ? em28xx_mt352_terratec_xs_init+0x120/0x120 [em28xx_dvb]
  [<d194438c>] dmx_ts_feed_start_filtering+0x4c/0xd0 [dvb_core]
  [<d1941d3d>] dvb_dmxdev_start_feed+0x9d/0xf0 [dvb_core]
  [<d1942f19>] dvb_dmxdev_filter_start+0x269/0x380 [dvb_core]
  [<d1942341>] ? dvb_dmxdev_add_pid+0x51/0xa0 [dvb_core]
  [<d19432f4>] dvb_demux_do_ioctl+0x164/0x550 [dvb_core]
  [<c10aca65>] ? path_put+0x15/0x20
  [<c10b0584>] ? do_last+0x1e4/0x7d0
  [<c117d497>] ? _copy_from_user+0x37/0x60
  [<d194131f>] dvb_usercopy+0xef/0x1a0 [dvb_core]
  [<c10b0f50>] ? do_filp_open+0x30/0x80
  [<d1941d90>] ? dvb_dmxdev_start_feed+0xf0/0xf0 [dvb_core]
  [<d1941da2>] dvb_demux_ioctl+0x12/0x20 [dvb_core]
  [<d1943190>] ? dvb_demux_release+0x160/0x160 [dvb_core]
  [<c10b2801>] vfs_ioctl+0x31/0x50
  [<c10b2ada>] do_vfs_ioctl+0xaa/0x500
  [<c10acdb3>] ? getname_flags+0xc3/0x130
  [<c10a27c8>] ? fd_install+0x48/0x60
  [<c10a294b>] ? do_sys_open+0x16b/0x1b0
  [<c1020000>] ? handle_vm86_fault+0x160/0x910
  [<c10b2f69>] sys_ioctl+0x39/0x70
  [<c13679d0>] sysenter_do_call+0x12/0x26
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:   90, btch:  15 usd:   0
active_anon:806 inactive_anon:10372 isolated_anon:0
  active_file:21713 inactive_file:22283 isolated_file:0
  unevictable:0 dirty:45 writeback:0 unstable:0
  free:994 slab_reclaimable:2584 slab_unreclaimable:1960
  mapped:1002 shmem:12 pagetables:409 bounce:0
DMA free:1100kB min:124kB low:152kB high:184kB active_anon:52kB 
inactive_anon:72kB active_file:6008kB inactive_file:8136kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB 
mlocked:0kB dirty:4kB writeback:0kB mapped:12kB shmem:0kB 
slab_reclaimable:436kB slab_unreclaimable:16kB kernel_stack:24kB 
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? no
lowmem_reserve[]: 0 237 237
Normal free:2756kB min:1904kB low:2380kB high:2856kB active_anon:3172kB 
inactive_anon:41416kB active_file:80844kB inactive_file:80996kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:242824kB 
mlocked:0kB dirty:176kB writeback:0kB mapped:3996kB shmem:48kB 
slab_reclaimable:9900kB slab_unreclaimable:7824kB kernel_stack:1904kB 
pagetables:1636kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 277*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 
0*2048kB 0*4096kB = 1108kB
Normal: 233*4kB 202*8kB 5*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 
0*1024kB 0*2048kB 0*4096kB = 2756kB
44485 total pagecache pages
476 pages in swap cache
Swap cache stats: add 1563, delete 1087, find 1624/1719
Free swap  = 485904kB
Total swap = 488340kB
65264 pages RAM
1946 pages reserved
49870 pages shared
36872 pages non-shared
unable to allocate 60160 bytes for transfer buffer 0

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 02/02/2012 18:28, Andy Furniss ha scritto:
> Hi.
> 
> I've got an old PIII 256M Ram PC that for years has served as an always
> on TV streamer/recorder and gateway/server box.
> 
> It had 2 PCI dvb cards until now I have a PCTV 290e so replaced one card
> with a USB2 PCI card.
> 
> At first the PCTV worked OK, but now I've been getting a page allocation
> failure. After this happens I can't use the PVTV until I re-plug it. It
> will then work, but the next day I am likely to get the same.
> 
> Free/meminfo (to my untrained eye) seem to show available mem.
> 
> mumudvb: page allocation failure: order:4, mode:0x80d0
> Pid: 15776, comm: mumudvb Not tainted 3.3.0-rc1 #6
> Call Trace:
>  [<c102a43b>] ? printk+0x1b/0x20
>  [<c107c353>] warn_alloc_failed+0xc3/0x130
>  [<c107dab7>] __alloc_pages_nodemask+0x447/0x580
>  [<c1006e58>] dma_generic_alloc_coherent+0x68/0xe0
>  [<c1255b41>] hcd_buffer_alloc+0x101/0x110
>  [<c1006df0>] ? dma_set_mask+0x50/0x50
>  [<c1248e35>] usb_alloc_coherent+0x25/0x30
>  [<d1dab1cc>] em28xx_init_isoc+0x10c/0x360 [em28xx]
>  [<d1dcb0ad>] em28xx_start_feed+0xad/0xe0 [em28xx_dvb]
>  [<d1dcca20>] ? em28xx_mt352_terratec_xs_init+0x120/0x120 [em28xx_dvb]
>  [<d194438c>] dmx_ts_feed_start_filtering+0x4c/0xd0 [dvb_core]
>  [<d1941d3d>] dvb_dmxdev_start_feed+0x9d/0xf0 [dvb_core]
>  [<d1942f19>] dvb_dmxdev_filter_start+0x269/0x380 [dvb_core]
>  [<d1942341>] ? dvb_dmxdev_add_pid+0x51/0xa0 [dvb_core]
>  [<d19432f4>] dvb_demux_do_ioctl+0x164/0x550 [dvb_core]
>  [<c10aca65>] ? path_put+0x15/0x20
>  [<c10b0584>] ? do_last+0x1e4/0x7d0
>  [<c117d497>] ? _copy_from_user+0x37/0x60
>  [<d194131f>] dvb_usercopy+0xef/0x1a0 [dvb_core]
>  [<c10b0f50>] ? do_filp_open+0x30/0x80
>  [<d1941d90>] ? dvb_dmxdev_start_feed+0xf0/0xf0 [dvb_core]
>  [<d1941da2>] dvb_demux_ioctl+0x12/0x20 [dvb_core]
>  [<d1943190>] ? dvb_demux_release+0x160/0x160 [dvb_core]
>  [<c10b2801>] vfs_ioctl+0x31/0x50
>  [<c10b2ada>] do_vfs_ioctl+0xaa/0x500
>  [<c10acdb3>] ? getname_flags+0xc3/0x130
>  [<c10a27c8>] ? fd_install+0x48/0x60
>  [<c10a294b>] ? do_sys_open+0x16b/0x1b0
>  [<c1020000>] ? handle_vm86_fault+0x160/0x910
>  [<c10b2f69>] sys_ioctl+0x39/0x70
>  [<c13679d0>] sysenter_do_call+0x12/0x26
> Mem-Info:
> DMA per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> Normal per-cpu:
> CPU    0: hi:   90, btch:  15 usd:   0
> active_anon:806 inactive_anon:10372 isolated_anon:0
>  active_file:21713 inactive_file:22283 isolated_file:0
>  unevictable:0 dirty:45 writeback:0 unstable:0
>  free:994 slab_reclaimable:2584 slab_unreclaimable:1960
>  mapped:1002 shmem:12 pagetables:409 bounce:0
> DMA free:1100kB min:124kB low:152kB high:184kB active_anon:52kB
> inactive_anon:72kB active_file:6008kB inactive_file:8136kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB
> mlocked:0kB dirty:4kB writeback:0kB mapped:12kB shmem:0kB
> slab_reclaimable:436kB slab_unreclaimable:16kB kernel_stack:24kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> lowmem_reserve[]: 0 237 237
> Normal free:2756kB min:1904kB low:2380kB high:2856kB active_anon:3172kB
> inactive_anon:41416kB active_file:80844kB inactive_file:80996kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:242824kB
> mlocked:0kB dirty:176kB writeback:0kB mapped:3996kB shmem:48kB
> slab_reclaimable:9900kB slab_unreclaimable:7824kB kernel_stack:1904kB
> pagetables:1636kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 277*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB = 1108kB
> Normal: 233*4kB 202*8kB 5*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 2756kB
> 44485 total pagecache pages
> 476 pages in swap cache
> Swap cache stats: add 1563, delete 1087, find 1624/1719
> Free swap  = 485904kB
> Total swap = 488340kB
> 65264 pages RAM
> 1946 pages reserved
> 49870 pages shared
> 36872 pages non-shared
> unable to allocate 60160 bytes for transfer buffer 0
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Andy,
I'm getting the same problem on a totally different system.
I'm using a set-top-box powered by a Broadcom mipsel SoC with 512 MB RAM.
When I use the Mediaplayer of the box for a few hours (which takes
almost all the RAM available in the system) and then switch back to DTT
TV, sometimes I get this crash:

usbtunerhelper: page allocation failure: order:4, mode:0x10d0
Call Trace:
[<8055f4e0>] dump_stack+0x8/0x34
[<8008ad00>] warn_alloc_failed+0xc4/0x144
[<8008b8c8>] __alloc_pages_nodemask+0x40c/0x678
[<8008bc30>] __get_free_pages+0x18/0x80
[<8001101c>] mips_dma_alloc_coherent+0x5c/0x114
[<e157ee9c>] em28xx_init_isoc+0x10c/0x3e4 [em28xx]
[<e159a71c>] em28xx_start_feed+0x12c/0x164 [em28xx_dvb]
[<803e4304>] dmx_ts_feed_start_filtering+0x5c/0x134
[<803e0664>] dvb_dmxdev_start_feed+0xd4/0x158
[<803e2c94>] dvb_demux_do_ioctl+0x578/0x654
[<803dfe00>] dvb_usercopy+0x88/0x204
[<800d6ef4>] do_vfs_ioctl+0xa0/0x6c0
[<800d7558>] sys_ioctl+0x44/0xa8
[<8000ecfc>] stack_done+0x20/0x40

Mem-Info:
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
active_anon:17666 inactive_anon:119 isolated_anon:0
 active_file:22491 inactive_file:21039 isolated_file:0
 unevictable:0 dirty:2 writeback:0 unstable:0
 free:3228 slab_reclaimable:1644 slab_unreclaimable:1955
 mapped:1415 shmem:169 pagetables:143 bounce:0
Normal free:12912kB min:2876kB low:3592kB high:4312kB
active_anon:70664kB inactive_anon:476kB active_file:89964kB
inactive_file:84156kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:518144kB mlocked:0kB dirty:8kB writeback:0kB
mapped:5660kB shmem:676kB slab_reclaimable:6576kB
slab_unreclaimable:7820kB kernel_stack:704kB pagetables:572kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
lowmem_reserve[]: 0 0
Normal: 1898*4kB 481*8kB 70*16kB 9*32kB 1*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 12912kB
43708 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
131072 pages RAM
58155 pages reserved
13909 pages shared
56534 pages non-shared
unable to allocate 48128 bytes for transfer buffer 0

I'm trying to reproduce the problem with another em28xx-dvb device to
see if it is not restricted to the PCTV 290e. Before the PCTV 290e, I
was using a different device with a driver based on the dvb-usb
framework, and I never observed similar crashes.

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Devin Heitmueller]

On Thu, Feb 2, 2012 at 2:02 PM, Gianluca Gennari <gennarone@gmail.com> wrote:
> I'm trying to reproduce the problem with another em28xx-dvb device to
> see if it is not restricted to the PCTV 290e. Before the PCTV 290e, I
> was using a different device with a driver based on the dvb-usb
> framework, and I never observed similar crashes.

On ARM based platforms it is very likely you will run into this issue
with most USB based tuners.  It's because over time there is memory
fragmentation that occurs which prevents being able to allocate large
enough chunks of coherent memory.

Making such a scenario work would require hacks to the driver code to
preallocate the memory in some form of static pool at system boot (or
perhaps at driver initialization), and then reuse that memory instead
of attempting to allocate it as needed.

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

Re: PCTV 290e page allocation failure

[Andy Furniss]

Gianluca Gennari wrote:

> Hi Andy,
> I'm getting the same problem on a totally different system.

Hi Gianluca,

Thanks for the reply - it's good to know it's not just me.

What kernel are you using?

I see someone else had problems with > 3.0, I've got a 3.08 built on 
this box, I'll try it out when I get a chance to reboot, though it took 
a couple of days to show on my current kernel.

Andy.

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 02/02/2012 20:07, Devin Heitmueller ha scritto:
> On Thu, Feb 2, 2012 at 2:02 PM, Gianluca Gennari <gennarone@gmail.com> wrote:
>> I'm trying to reproduce the problem with another em28xx-dvb device to
>> see if it is not restricted to the PCTV 290e. Before the PCTV 290e, I
>> was using a different device with a driver based on the dvb-usb
>> framework, and I never observed similar crashes.
> 
> On ARM based platforms it is very likely you will run into this issue
> with most USB based tuners.  It's because over time there is memory
> fragmentation that occurs which prevents being able to allocate large
> enough chunks of coherent memory.
> 
> Making such a scenario work would require hacks to the driver code to
> preallocate the memory in some form of static pool at system boot (or
> perhaps at driver initialization), and then reuse that memory instead
> of attempting to allocate it as needed.
> 
> Devin
> 

Hi Devin,
thanks for the explanation. The CPU is MIPS based (not ARM) but I guess
there is not much of a difference from this point of view.
As I mentioned in my first reply, I never had this kind of errors when I
was using a dvb-usb USB stick. Now I'm trying to replicate the problem
with a Terratec Hybrid XS (em28xx-dvb + zl10353 + xc2028), and so far
I've stressed it for a few hours without problems. We will see in a day
or two if I can make it fail in the same way.

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 02/02/2012 20:56, Andy Furniss ha scritto:
> Gianluca Gennari wrote:
> 
>> Hi Andy,
>> I'm getting the same problem on a totally different system.
> 
> Hi Gianluca,
> 
> Thanks for the reply - it's good to know it's not just me.
> 
> What kernel are you using?
> 
> I see someone else had problems with > 3.0, I've got a 3.08 built on
> this box, I'll try it out when I get a chance to reboot, though it took
> a couple of days to show on my current kernel.
> 
> Andy.
> 

Hi Andy,
I'm running 3.1.0 but I back-ported a few patches from 3.2.0 to update
the PCTV 290e driver to the latest version.
In the past months I run 2.6.18/2.6.31/3.0.3 before buying the PCTV
290e, but I never had this problem with the old dvb-usb stick.

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Devin Heitmueller]

On Thu, Feb 2, 2012 at 6:06 PM, Gianluca Gennari <gennarone@gmail.com> wrote:
> Il 02/02/2012 20:07, Devin Heitmueller ha scritto:
> Hi Devin,
> thanks for the explanation. The CPU is MIPS based (not ARM) but I guess
> there is not much of a difference from this point of view.
> As I mentioned in my first reply, I never had this kind of errors when I
> was using a dvb-usb USB stick. Now I'm trying to replicate the problem
> with a Terratec Hybrid XS (em28xx-dvb + zl10353 + xc2028), and so far
> I've stressed it for a few hours without problems. We will see in a day
> or two if I can make it fail in the same way.

I'm pretty sure this will happen under MIPS as well.  That said, you
will typically hit this condition if you stop streaming and then
restart it several hours into operation.  In other words, make sure
you're not just watching/streaming video for a few hours and thinking
you're stressing the particular use case.  You need to stop/start to
hit it.

I haven't looked that closely at dvb_usb's memory allocation strategy.
 Perhaps it allocates the memory up front, or perhaps it doesn't
demand coherent memory (something which will work on x86 and maybe
MIPS, but will cause an immediate panic on ARM).

I've run into this issue myself on an embedded target with em28xx and
ARM.  I plan on hacking a fix to statically allocate the buffers at
driver init, but I cannot imagine that being a change that would be
accepted into the upstream kernel.

It probably makes sense to figure out whether MIPS requires coherent
memory like ARM does.  If it doesn't then you can probably just hack
your copy of the em28xx driver to not ask for coherent memory.  If it
does require coherent memory, then you'll probably need to allocate
the memory up front.

Cheers,

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 03/02/2012 00:12, Devin Heitmueller ha scritto:
> On Thu, Feb 2, 2012 at 6:06 PM, Gianluca Gennari <gennarone@gmail.com> wrote:
>> Il 02/02/2012 20:07, Devin Heitmueller ha scritto:
>> Hi Devin,
>> thanks for the explanation. The CPU is MIPS based (not ARM) but I guess
>> there is not much of a difference from this point of view.
>> As I mentioned in my first reply, I never had this kind of errors when I
>> was using a dvb-usb USB stick. Now I'm trying to replicate the problem
>> with a Terratec Hybrid XS (em28xx-dvb + zl10353 + xc2028), and so far
>> I've stressed it for a few hours without problems. We will see in a day
>> or two if I can make it fail in the same way.
> 
> I'm pretty sure this will happen under MIPS as well.  That said, you
> will typically hit this condition if you stop streaming and then
> restart it several hours into operation.  In other words, make sure
> you're not just watching/streaming video for a few hours and thinking
> you're stressing the particular use case.  You need to stop/start to
> hit it.

Yes, I've been switching between the mediaplayer (playing some 1080p mkv
file to stress the memory) and the USB tuner, but so far so good.
But I need to run the test longer to draw some conclusion.

> I haven't looked that closely at dvb_usb's memory allocation strategy.
>  Perhaps it allocates the memory up front, or perhaps it doesn't
> demand coherent memory (something which will work on x86 and maybe
> MIPS, but will cause an immediate panic on ARM).
> 
> I've run into this issue myself on an embedded target with em28xx and
> ARM.  I plan on hacking a fix to statically allocate the buffers at
> driver init, but I cannot imagine that being a change that would be
> accepted into the upstream kernel.

If you have some patch that you want to share, I will be happy to test it.

> It probably makes sense to figure out whether MIPS requires coherent
> memory like ARM does.  If it doesn't then you can probably just hack
> your copy of the em28xx driver to not ask for coherent memory.  If it
> does require coherent memory, then you'll probably need to allocate
> the memory up front.
> 

Interesting suggestion. I have really no idea if MIPS really requires
coherent memory in this case. I may try to hack it and see what happens.

Thank you very much for the detailed explanation.

Regards,
Gianluca

> Cheers,
> 
> Devin
> 

Re: PCTV 290e page allocation failure

[Andy Furniss]

Gianluca Gennari wrote:

>> What kernel are you using?
>>
>> I see someone else had problems with>  3.0, I've got a 3.08 built on
>> this box, I'll try it out when I get a chance to reboot, though it took
>> a couple of days to show on my current kernel.
>>
>> Andy.
>>
>
> Hi Andy,
> I'm running 3.1.0 but I back-ported a few patches from 3.2.0 to update
> the PCTV 290e driver to the latest version.
> In the past months I run 2.6.18/2.6.31/3.0.3 before buying the PCTV
> 290e, but I never had this problem with the old dvb-usb stick.

Hi,

I tried my 3.08 but changed back as I was getting corrupted HD streams.

Maybe because the config for that kernel was no SMP and no preemption.

I did do lots of cat /proc/buddyinfo and echo m > /proc/sysrq-trigger 
and it looks like having lots of files open is my problem - but I didn't 
run long enough to provoke a fail.

It seems even if the above commands show no continuous DMA above 16k 
when you actually try and use it the kernel defrags so it works and the 
output will then show larger chunks available for a while.

When I had 2xPCI running it was mainly on 2.6.26 and I was also using 
legacy IDE - I wonder if that behaved differently with 00s of open files 
- or maybe it's just that PCIs (remaining one is cx88) just don't ask 
for big 64k DMA buffers.

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 03/02/2012 12:04, Andy Furniss ha scritto:
> Gianluca Gennari wrote:
> 
>>> What kernel are you using?
>>>
>>> I see someone else had problems with>  3.0, I've got a 3.08 built on
>>> this box, I'll try it out when I get a chance to reboot, though it took
>>> a couple of days to show on my current kernel.
>>>
>>> Andy.
>>>
>>
>> Hi Andy,
>> I'm running 3.1.0 but I back-ported a few patches from 3.2.0 to update
>> the PCTV 290e driver to the latest version.
>> In the past months I run 2.6.18/2.6.31/3.0.3 before buying the PCTV
>> 290e, but I never had this problem with the old dvb-usb stick.
> 
> Hi,
> 
> I tried my 3.08 but changed back as I was getting corrupted HD streams.
> 
> Maybe because the config for that kernel was no SMP and no preemption.
> 
> I did do lots of cat /proc/buddyinfo and echo m > /proc/sysrq-trigger
> and it looks like having lots of files open is my problem - but I didn't
> run long enough to provoke a fail.
> 
> It seems even if the above commands show no continuous DMA above 16k
> when you actually try and use it the kernel defrags so it works and the
> output will then show larger chunks available for a while.
> 
> When I had 2xPCI running it was mainly on 2.6.26 and I was also using
> legacy IDE - I wonder if that behaved differently with 00s of open files
> - or maybe it's just that PCIs (remaining one is cx88) just don't ask
> for big 64k DMA buffers.
> 
> 

Hi,
I was able to reproduce the crash with the Terratec Hybrid card after
about 1 day of use (switching between the mediaplayer and the DVB-T tuner):

usbtunerhelper: page allocation failure: order:4, mode:0x10d0
Call Trace:
[<80550be0>] dump_stack+0x8/0x34
[<8008aca0>] warn_alloc_failed+0xc4/0x144
[<8008b868>] __alloc_pages_nodemask+0x40c/0x678
[<8008bbd0>] __get_free_pages+0x18/0x80
[<8001101c>] mips_dma_alloc_coherent+0x5c/0x114
[<e16a7e9c>] em28xx_init_isoc+0x10c/0x3e4 [em28xx]
[<e174b71c>] em28xx_start_feed+0x12c/0x164 [em28xx_dvb]
[<803ce924>] dmx_ts_feed_start_filtering+0x5c/0x134
[<803cac84>] dvb_dmxdev_start_feed+0xd4/0x158
[<803cd2b4>] dvb_demux_do_ioctl+0x578/0x654
[<803ca420>] dvb_usercopy+0x88/0x204
[<800d6e94>] do_vfs_ioctl+0xa0/0x6c0
[<800d74f8>] sys_ioctl+0x44/0xa8
[<8000ecfc>] stack_done+0x20/0x40

Mem-Info:
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
active_anon:16570 inactive_anon:27 isolated_anon:0
 active_file:21560 inactive_file:21546 isolated_file:0
 unevictable:0 dirty:2 writeback:0 unstable:0
 free:3996 slab_reclaimable:2042 slab_unreclaimable:2124
 mapped:2392 shmem:31 pagetables:126 bounce:0
Normal free:15984kB min:2876kB low:3592kB high:4312kB
active_anon:66280kB inactive_anon:108kB active_file:86240kB
inactive_file:86184kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:518144kB mlocked:0kB dirty:8kB writeback:0kB
mapped:9568kB shmem:124kB slab_reclaimable:8168kB
slab_unreclaimable:8496kB kernel_stack:712kB pagetables:504kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
lowmem_reserve[]: 0 0
Normal: 2556*4kB 540*8kB 80*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 15984kB
43141 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
131072 pages RAM
58145 pages reserved
10852 pages shared
59573 pages non-shared
unable to allocate 36096 bytes for transfer buffer 0

free:
             total         used         free       shared      buffers
Mem:        291708       274656        17052            0         7592
-/+ buffers:             267064        24644
Swap:            0            0            0

(the box has 512 MB but about 220 MB are reserved for the framebuffer)

So there is no doubt it's a generic problem (as Devin already pointed
out) and not restricted to the PCTV 290e.

I will now try to reproduce the issue with a dvb-usb driver.

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Hi all,
I compared the memory allocation strategies for isoc transfers of the
dvb-usb and em28xx drivers.

There are 3 main differences:

1) dvb-usb drivers allocate the URBs when the device is connected, and
they are never freed/reallocated until the device is disconnected; on
the other hand, the em28xx driver allocates the URBs only when the data
starts streaming and frees them when the stream stops, so the URBs are
freed/reallocated every time the user zaps to a new channel;

2) dvb-usb drivers typically allocate 10 URBs each with a 4k buffer (but
the exact size of the buffer is board dependent); instead, em28xx
allocates 5 URBs with buffers of size 64xMaxPacketSize (which is 940
byte for the PCTV 290e).

This means a typical dvb-usb driver uses about 40k of coherent memory,
while the PCTV 290e takes about 300k! And this 300k of coherent memory
are freed/reallocated each time the user selects a new channel.

I played a bit with the size of the buffers; I found out that both the
PCTV 290e and the Terratec Hybrid XS work perfectly fine with 4k
buffers, just like the usb-dvb drivers. So the PCTV 290e only needs 20k
of coherent memory instead of the 300k currently allocated (this is
equivalent to set EM28XX_DVB_MAX_PACKETS to just 4 instead of 64).

Also, I prepared a proof-of-concept patch to mimic the usb-dvb URB
management; this means the URBs are allocated when the USB device is
probed, and are freed when the device is disconnected (the patch code
checks for changes in the requested buffer size, but this can never
happen in digital mode).

Also the patch adds a module parameter "transfer_buf_size" that allows
to override the default buffer size.

For example:

modprobe em28xx transfer_buf_size=4096

will force the URB buffer size to 4k instead of 64*MaxPacketSize.

I've tested the patch (with 4k buffers) with both my em28xx sticks and
both work perfectly fine on my PC as well as on my MIPS set-top-box.
Analog mode is not tested.

If you think there is any chance to have some of this changes in the
upstream kernel, I can spend some more time to clean up the code.

Regards,
Gianluca

---
 drivers/media/video/em28xx/em28xx-cards.c |   12 ++-
 drivers/media/video/em28xx/em28xx-core.c  |  167
+++++++++++++++++------------
 drivers/media/video/em28xx/em28xx-dvb.c   |    7 +-
 drivers/media/video/em28xx/em28xx-video.c |    5 +-
 drivers/media/video/em28xx/em28xx.h       |   10 ++-
 5 files changed, 122 insertions(+), 79 deletions(-)

diff --git a/drivers/media/video/em28xx/em28xx-cards.c
b/drivers/media/video/em28xx/em28xx-cards.c
index 2aa772a..3fb7744 100644
--- a/drivers/media/video/em28xx/em28xx-cards.c
+++ b/drivers/media/video/em28xx/em28xx-cards.c
@@ -3321,6 +3321,14 @@ static int em28xx_usb_probe(struct usb_interface
*interface,
 	 */
 	em28xx_init_extension(dev);

+	if (has_dvb && dev->dvb_max_pkt_size > 0) {
+		/* pre-alloc isoc transfer buffers */
+		em28xx_init_isoc(dev, EM28XX_DVB_MAX_PACKETS,
+			EM28XX_DVB_NUM_BUFS, dev->dvb_max_pkt_size,
+			false, NULL);
+		dev->isoc_ctl.preallocated_transfer_bufs = true;
+	}
+
 	return 0;

 unlock_and_free:
@@ -3372,6 +3380,9 @@ static void em28xx_usb_disconnect(struct
usb_interface *interface)

 	v4l2_device_disconnect(&dev->v4l2_dev);

+	dev->isoc_ctl.preallocated_transfer_bufs = false;
+	em28xx_uninit_isoc(dev, true);
+
 	if (dev->users) {
 		em28xx_warn
 		    ("device %s is open! Deregistration and memory "
@@ -3379,7 +3390,6 @@ static void em28xx_usb_disconnect(struct
usb_interface *interface)
 		     video_device_node_name(dev->vdev));

 		dev->state |= DEV_MISCONFIGURED;
-		em28xx_uninit_isoc(dev);
 		dev->state |= DEV_DISCONNECTED;
 		wake_up_interruptible(&dev->wait_frame);
 		wake_up_interruptible(&dev->wait_stream);
diff --git a/drivers/media/video/em28xx/em28xx-core.c
b/drivers/media/video/em28xx/em28xx-core.c
index 0aacc96..9c84f48 100644
--- a/drivers/media/video/em28xx/em28xx-core.c
+++ b/drivers/media/video/em28xx/em28xx-core.c
@@ -59,6 +59,10 @@ static unsigned int disable_vbi;
 module_param(disable_vbi, int, 0644);
 MODULE_PARM_DESC(disable_vbi, "disable vbi support");

+static unsigned int transfer_buf_size;
+module_param(transfer_buf_size, int, 0644);
+MODULE_PARM_DESC(transfer_buf_size, "override USB isoc transfer buffer
size");
+
 /* FIXME */
 #define em28xx_isocdbg(fmt, arg...) do {\
 	if (core_debug) \
@@ -961,14 +965,18 @@ static void em28xx_irq_callback(struct urb *urb)
 /*
  * Stop and Deallocate URBs
  */
-void em28xx_uninit_isoc(struct em28xx *dev)
+void em28xx_uninit_isoc(struct em28xx *dev, bool start)
 {
 	struct urb *urb;
 	int i;

 	em28xx_isocdbg("em28xx: called em28xx_uninit_isoc\n");

+	if (dev->isoc_ctl.preallocated_transfer_bufs)
+		return;
+
 	dev->isoc_ctl.nfields = -1;
+	printk ("free %d isoc transfer buffers\n", dev->isoc_ctl.num_bufs);
 	for (i = 0; i < dev->isoc_ctl.num_bufs; i++) {
 		urb = dev->isoc_ctl.urb[i];
 		if (urb) {
@@ -996,7 +1004,8 @@ void em28xx_uninit_isoc(struct em28xx *dev)
 	dev->isoc_ctl.transfer_buffer = NULL;
 	dev->isoc_ctl.num_bufs = 0;

-	em28xx_capture_start(dev, 0);
+	if (start)
+		em28xx_capture_start(dev, 0);
 }
 EXPORT_SYMBOL_GPL(em28xx_uninit_isoc);

@@ -1004,7 +1013,7 @@ EXPORT_SYMBOL_GPL(em28xx_uninit_isoc);
  * Allocate URBs and start IRQ
  */
 int em28xx_init_isoc(struct em28xx *dev, int max_packets,
-		     int num_bufs, int max_pkt_size,
+		     int num_bufs, int max_pkt_size, bool start,
 		     int (*isoc_copy) (struct em28xx *dev, struct urb *urb))
 {
 	struct em28xx_dmaqueue *dma_q = &dev->vidq;
@@ -1017,78 +1026,99 @@ int em28xx_init_isoc(struct em28xx *dev, int
max_packets,

 	em28xx_isocdbg("em28xx: called em28xx_prepare_isoc\n");

-	/* De-allocates all pending stuff */
-	em28xx_uninit_isoc(dev);
-
-	dev->isoc_ctl.isoc_copy = isoc_copy;
-	dev->isoc_ctl.num_bufs = num_bufs;
-
-	dev->isoc_ctl.urb = kzalloc(sizeof(void *)*num_bufs,  GFP_KERNEL);
-	if (!dev->isoc_ctl.urb) {
-		em28xx_errdev("cannot alloc memory for usb buffers\n");
-		return -ENOMEM;
-	}
-
-	dev->isoc_ctl.transfer_buffer = kzalloc(sizeof(void *)*num_bufs,
-					      GFP_KERNEL);
-	if (!dev->isoc_ctl.transfer_buffer) {
-		em28xx_errdev("cannot allocate memory for usb transfer\n");
-		kfree(dev->isoc_ctl.urb);
-		return -ENOMEM;
-	}
-
-	dev->isoc_ctl.max_pkt_size = max_pkt_size;
-	dev->isoc_ctl.vid_buf = NULL;
-	dev->isoc_ctl.vbi_buf = NULL;
-
-	sb_size = max_packets * dev->isoc_ctl.max_pkt_size;
-
-	/* allocate urbs and transfer buffers */
-	for (i = 0; i < dev->isoc_ctl.num_bufs; i++) {
-		urb = usb_alloc_urb(max_packets, GFP_KERNEL);
-		if (!urb) {
-			em28xx_err("cannot alloc isoc_ctl.urb %i\n", i);
-			em28xx_uninit_isoc(dev);
+	if (transfer_buf_size && max_pkt_size > 0) {
+		sb_size = transfer_buf_size;
+		max_packets = sb_size / max_pkt_size;
+		printk ("overridden sb_size:%d and max_packets:%d\n",
+			sb_size, max_packets);
+	} else
+		sb_size = max_packets * max_pkt_size;
+
+	if (dev->isoc_ctl.transfer_buffer == NULL
+	    || dev->isoc_ctl.num_bufs != num_bufs
+	    || dev->isoc_ctl.max_pkt_size != max_pkt_size
+	    || (dev->isoc_ctl.num_bufs > 0
+	     && dev->isoc_ctl.urb[0]->number_of_packets != max_packets)) {
+		/* De-allocates all pending stuff */
+		dev->isoc_ctl.preallocated_transfer_bufs = false;
+		em28xx_uninit_isoc(dev, start);
+
+		dev->isoc_ctl.num_bufs = num_bufs;
+
+		dev->isoc_ctl.urb = kzalloc(sizeof(void *)*num_bufs,  GFP_KERNEL);
+		if (!dev->isoc_ctl.urb) {
+			em28xx_errdev("cannot alloc memory for usb buffers\n");
 			return -ENOMEM;
 		}
-		dev->isoc_ctl.urb[i] = urb;
-
-		dev->isoc_ctl.transfer_buffer[i] = usb_alloc_coherent(dev->udev,
-			sb_size, GFP_KERNEL, &urb->transfer_dma);
-		if (!dev->isoc_ctl.transfer_buffer[i]) {
-			em28xx_err("unable to allocate %i bytes for transfer"
-					" buffer %i%s\n",
-					sb_size, i,
-					in_interrupt() ? " while in int" : "");
-			em28xx_uninit_isoc(dev);
+
+		dev->isoc_ctl.transfer_buffer = kzalloc(sizeof(void *)*num_bufs,
+									GFP_KERNEL);
+		if (!dev->isoc_ctl.transfer_buffer) {
+			em28xx_errdev("cannot allocate memory for usb transfer\n");
+			kfree(dev->isoc_ctl.urb);
 			return -ENOMEM;
 		}
-		memset(dev->isoc_ctl.transfer_buffer[i], 0, sb_size);

-		/* FIXME: this is a hack - should be
-			'desc.bEndpointAddress & USB_ENDPOINT_NUMBER_MASK'
-			should also be using 'desc.bInterval'
-		 */
-		pipe = usb_rcvisocpipe(dev->udev,
-				       dev->mode == EM28XX_ANALOG_MODE ?
-				       EM28XX_EP_ANALOG : EM28XX_EP_DIGITAL);
-
-		usb_fill_int_urb(urb, dev->udev, pipe,
-				 dev->isoc_ctl.transfer_buffer[i], sb_size,
-				 em28xx_irq_callback, dev, 1);
-
-		urb->number_of_packets = max_packets;
-		urb->transfer_flags = URB_ISO_ASAP | URB_NO_TRANSFER_DMA_MAP;
-
-		k = 0;
-		for (j = 0; j < max_packets; j++) {
-			urb->iso_frame_desc[j].offset = k;
-			urb->iso_frame_desc[j].length =
-						dev->isoc_ctl.max_pkt_size;
-			k += dev->isoc_ctl.max_pkt_size;
+		dev->isoc_ctl.max_pkt_size = max_pkt_size;
+		dev->isoc_ctl.vid_buf = NULL;
+		dev->isoc_ctl.vbi_buf = NULL;
+
+		printk ("alloc %d buffers of %d bytes (each containing %d packets of
%d bytes)\n",
+			dev->isoc_ctl.num_bufs, sb_size, max_packets,
dev->isoc_ctl.max_pkt_size);
+		/* allocate urbs and transfer buffers */
+		for (i = 0; i < dev->isoc_ctl.num_bufs; i++) {
+			urb = usb_alloc_urb(max_packets, GFP_KERNEL);
+			if (!urb) {
+				em28xx_err("cannot alloc isoc_ctl.urb %i\n", i);
+				em28xx_uninit_isoc(dev, start);
+				return -ENOMEM;
+			}
+			dev->isoc_ctl.urb[i] = urb;
+
+			dev->isoc_ctl.transfer_buffer[i] = usb_alloc_coherent(dev->udev,
+				sb_size, GFP_KERNEL, &urb->transfer_dma);
+			if (!dev->isoc_ctl.transfer_buffer[i]) {
+				em28xx_err("unable to allocate %i bytes for transfer"
+						" buffer %i%s\n",
+						sb_size, i,
+						in_interrupt() ? " while in int" : "");
+				em28xx_uninit_isoc(dev, start);
+				return -ENOMEM;
+			}
+			memset(dev->isoc_ctl.transfer_buffer[i], 0, sb_size);
+
+			/* FIXME: this is a hack - should be
+				'desc.bEndpointAddress & USB_ENDPOINT_NUMBER_MASK'
+				should also be using 'desc.bInterval'
+			*/
+			pipe = usb_rcvisocpipe(dev->udev,
+								dev->mode == EM28XX_ANALOG_MODE ?
+								EM28XX_EP_ANALOG : EM28XX_EP_DIGITAL);
+
+			usb_fill_int_urb(urb, dev->udev, pipe,
+					dev->isoc_ctl.transfer_buffer[i], sb_size,
+					em28xx_irq_callback, dev, 1);
+
+			urb->number_of_packets = max_packets;
+			urb->transfer_flags = URB_ISO_ASAP | URB_NO_TRANSFER_DMA_MAP;
+
+			k = 0;
+			for (j = 0; j < max_packets; j++) {
+				urb->iso_frame_desc[j].offset = k;
+				urb->iso_frame_desc[j].length =
+							dev->isoc_ctl.max_pkt_size;
+				k += dev->isoc_ctl.max_pkt_size;
+			}
 		}
+	} else {
+		printk ("no need to realloc isoc transfer buffers (size not changed)\n");
 	}

+	dev->isoc_ctl.isoc_copy = isoc_copy;
+
+	if (!start)
+		return 0;
+
 	init_waitqueue_head(&dma_q->wq);
 	init_waitqueue_head(&vbi_dma_q->wq);

@@ -1100,7 +1130,8 @@ int em28xx_init_isoc(struct em28xx *dev, int
max_packets,
 		if (rc) {
 			em28xx_err("submit of urb %i failed (error=%i)\n", i,
 				   rc);
-			em28xx_uninit_isoc(dev);
+			dev->isoc_ctl.preallocated_transfer_bufs = false;
+			em28xx_uninit_isoc(dev, start);
 			return rc;
 		}
 	}
diff --git a/drivers/media/video/em28xx/em28xx-dvb.c
b/drivers/media/video/em28xx/em28xx-dvb.c
index aabbf48..13272ad 100644
--- a/drivers/media/video/em28xx/em28xx-dvb.c
+++ b/drivers/media/video/em28xx/em28xx-dvb.c
@@ -61,9 +61,6 @@ if (debug >= level) 						\
 	printk(KERN_DEBUG "%s/2-dvb: " fmt, dev->name, ## arg);	\
 } while (0)

-#define EM28XX_DVB_NUM_BUFS 5
-#define EM28XX_DVB_MAX_PACKETS 64
-
 struct em28xx_dvb {
 	struct dvb_frontend        *fe[2];

@@ -178,15 +175,13 @@ static int em28xx_start_streaming(struct
em28xx_dvb *dvb)

 	return em28xx_init_isoc(dev, EM28XX_DVB_MAX_PACKETS,
 				EM28XX_DVB_NUM_BUFS, max_dvb_packet_size,
-				em28xx_dvb_isoc_copy);
+				true, em28xx_dvb_isoc_copy);
 }

 static int em28xx_stop_streaming(struct em28xx_dvb *dvb)
 {
 	struct em28xx *dev = dvb->adapter.priv;

-	em28xx_uninit_isoc(dev);
-
 	em28xx_set_mode(dev, EM28XX_SUSPEND);

 	return 0;
diff --git a/drivers/media/video/em28xx/em28xx-video.c
b/drivers/media/video/em28xx/em28xx-video.c
index 613300b..35b4096 100644
--- a/drivers/media/video/em28xx/em28xx-video.c
+++ b/drivers/media/video/em28xx/em28xx-video.c
@@ -764,15 +764,18 @@ buffer_prepare(struct videobuf_queue *vq, struct
videobuf_buffer *vb,
 		urb_init = 1;

 	if (urb_init) {
+		dev->isoc_ctl.preallocated_transfer_bufs = false;
 		if (em28xx_vbi_supported(dev) == 1)
 			rc = em28xx_init_isoc(dev, EM28XX_NUM_PACKETS,
 					      EM28XX_NUM_BUFS,
 					      dev->max_pkt_size,
+					      true,
 					      em28xx_isoc_copy_vbi);
 		else
 			rc = em28xx_init_isoc(dev, EM28XX_NUM_PACKETS,
 					      EM28XX_NUM_BUFS,
 					      dev->max_pkt_size,
+					      true,
 					      em28xx_isoc_copy);
 		if (rc < 0)
 			goto fail;
@@ -2267,7 +2270,7 @@ static int em28xx_v4l2_close(struct file *filp)
 		v4l2_device_call_all(&dev->v4l2_dev, 0, core, s_power, 0);

 		/* do this before setting alternate! */
-		em28xx_uninit_isoc(dev);
+		em28xx_uninit_isoc(dev, true);
 		em28xx_set_mode(dev, EM28XX_SUSPEND);

 		/* set alternate 0 */
diff --git a/drivers/media/video/em28xx/em28xx.h
b/drivers/media/video/em28xx/em28xx.h
index 22e252b..fcbd60a 100644
--- a/drivers/media/video/em28xx/em28xx.h
+++ b/drivers/media/video/em28xx/em28xx.h
@@ -151,12 +151,14 @@

 /* number of buffers for isoc transfers */
 #define EM28XX_NUM_BUFS 5
+#define EM28XX_DVB_NUM_BUFS 5

 /* number of packets for each buffer
    windows requests only 64 packets .. so we better do the same
    this is what I found out for all alternate numbers there!
  */
 #define EM28XX_NUM_PACKETS 64
+#define EM28XX_DVB_MAX_PACKETS 64

 #define EM28XX_INTERLACED_DEFAULT 1

@@ -228,9 +230,11 @@ struct em28xx_usb_isoc_ctl {
 		/* Stores the number of received fields */
 	int				nfields;

+		/* Signals if the transfer buffers were preallocated */
+	bool preallocated_transfer_bufs;
+
 		/* isoc urb callback */
 	int (*isoc_copy) (struct em28xx *dev, struct urb *urb);
-
 };

 /* Struct to enumberate video formats */
@@ -677,9 +681,9 @@ int em28xx_set_outfmt(struct em28xx *dev);
 int em28xx_resolution_set(struct em28xx *dev);
 int em28xx_set_alternate(struct em28xx *dev);
 int em28xx_init_isoc(struct em28xx *dev, int max_packets,
-		     int num_bufs, int max_pkt_size,
+		     int num_bufs, int max_pkt_size, bool start,
 		     int (*isoc_copy) (struct em28xx *dev, struct urb *urb));
-void em28xx_uninit_isoc(struct em28xx *dev);
+void em28xx_uninit_isoc(struct em28xx *dev, bool start);
 int em28xx_isoc_dvb_max_packetsize(struct em28xx *dev);
 int em28xx_set_mode(struct em28xx *dev, enum em28xx_mode set_mode);
 int em28xx_gpio_set(struct em28xx *dev, struct em28xx_reg_seq *gpio);
-- 
1.7.0.4

Re: PCTV 290e page allocation failure

[Andy Furniss]

Gianluca Gennari wrote:

> If you think there is any chance to have some of this changes in the
> upstream kernel, I can spend some more time to clean up the code.

Great work, I for one hope this can get.

Re: PCTV 290e page allocation failure

[Andy Furniss]

Andy Furniss wrote:

> At first the PCTV worked OK, but now I've been getting a page allocation
> failure. After this happens I can't use the PVTV until I re-plug it.

Further testing on this shows that even if I free up lots of memory with

sync;echo 3 > /proc/sys/vm/drop_caches giving -

DMA: 55*4kB 55*8kB 57*16kB 53*32kB 40*64kB 26*128kB 9*256kB 4*512kB 
2*1024kB 0*2048kB 0*4096kB = 15556kB
Normal: 2996*4kB 2169*8kB 1616*16kB 1021*32kB 596*64kB 221*128kB 
45*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 166840kB

It will still fail if it has already failed and not been replugged.

It's not failing to allocate - it's just not trying to allocate AFAICT , 
which I guess counts as a bug?

Re: PCTV 290e page allocation failure

[Devin Heitmueller]

On Tue, Feb 7, 2012 at 5:44 AM, Gianluca Gennari <gennarone@gmail.com> wrote:
> 1) dvb-usb drivers allocate the URBs when the device is connected, and
> they are never freed/reallocated until the device is disconnected; on
> the other hand, the em28xx driver allocates the URBs only when the data
> starts streaming and frees them when the stream stops, so the URBs are
> freed/reallocated every time the user zaps to a new channel;

Correct, the current strategy is to optimized to minimize memory use
when the device is not active, as opposed to tying up the memory when
it's not in use (obviously at the cost of the memory possibly not
being available on certain ARM/MIPS platforms).

> 2) dvb-usb drivers typically allocate 10 URBs each with a 4k buffer (but
> the exact size of the buffer is board dependent); instead, em28xx
> allocates 5 URBs with buffers of size 64xMaxPacketSize (which is 940
> byte for the PCTV 290e).
>
> This means a typical dvb-usb driver uses about 40k of coherent memory,
> while the PCTV 290e takes about 300k! And this 300k of coherent memory
> are freed/reallocated each time the user selects a new channel.
>
> I played a bit with the size of the buffers; I found out that both the
> PCTV 290e and the Terratec Hybrid XS work perfectly fine with 4k
> buffers, just like the usb-dvb drivers. So the PCTV 290e only needs 20k
> of coherent memory instead of the 300k currently allocated (this is
> equivalent to set EM28XX_DVB_MAX_PACKETS to just 4 instead of 64).

This one is a bit harder to speculate on.  Are you actually capturing
all the packets and ensuring there are no packets being dropped (e.g.
looking for discontinuities)?  Have you tried it with modulation types
that are high bandwidth?  Have you tried capturing an entire stream
with PID filtering disabled?  The change you described may "appear" to
be working when in fact it's only working for a subset of real-world
use cases.

Also, the buffer management may be appropriate for the em2874/em2884,
but be broken for other devices such as the em288[0123].  Testing
would be required to ensure it's not introducing regressions.  In
particular, the 2874/2884 had architecture changes which may result in
differences in behavior (the register map is significantly different,
for example).

> Also, I prepared a proof-of-concept patch to mimic the usb-dvb URB
> management; this means the URBs are allocated when the USB device is
> probed, and are freed when the device is disconnected (the patch code
> checks for changes in the requested buffer size, but this can never
> happen in digital mode).

I don't have any specific problem with such a change assuming it is
properly vetted against other devices.

> I've tested the patch (with 4k buffers) with both my em28xx sticks and
> both work perfectly fine on my PC as well as on my MIPS set-top-box.
> Analog mode is not tested.

Again, the big question here is surrounding your testing methodology.
If you could expand on how you're testing, that would be helpful in
assessing how well the patch will really work.

Thanks,

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 07/02/2012 16:18, Devin Heitmueller ha scritto:
> On Tue, Feb 7, 2012 at 5:44 AM, Gianluca Gennari <gennarone@gmail.com> wrote:
>> 1) dvb-usb drivers allocate the URBs when the device is connected, and
>> they are never freed/reallocated until the device is disconnected; on
>> the other hand, the em28xx driver allocates the URBs only when the data
>> starts streaming and frees them when the stream stops, so the URBs are
>> freed/reallocated every time the user zaps to a new channel;
> 
> Correct, the current strategy is to optimized to minimize memory use
> when the device is not active, as opposed to tying up the memory when
> it's not in use (obviously at the cost of the memory possibly not
> being available on certain ARM/MIPS platforms).

Ok, that's clear.

>> 2) dvb-usb drivers typically allocate 10 URBs each with a 4k buffer (but
>> the exact size of the buffer is board dependent); instead, em28xx
>> allocates 5 URBs with buffers of size 64xMaxPacketSize (which is 940
>> byte for the PCTV 290e).
>>
>> This means a typical dvb-usb driver uses about 40k of coherent memory,
>> while the PCTV 290e takes about 300k! And this 300k of coherent memory
>> are freed/reallocated each time the user selects a new channel.
>>
>> I played a bit with the size of the buffers; I found out that both the
>> PCTV 290e and the Terratec Hybrid XS work perfectly fine with 4k
>> buffers, just like the usb-dvb drivers. So the PCTV 290e only needs 20k
>> of coherent memory instead of the 300k currently allocated (this is
>> equivalent to set EM28XX_DVB_MAX_PACKETS to just 4 instead of 64).
> 
> This one is a bit harder to speculate on.  Are you actually capturing
> all the packets and ensuring there are no packets being dropped (e.g.
> looking for discontinuities)?  Have you tried it with modulation types
> that are high bandwidth?  Have you tried capturing an entire stream
> with PID filtering disabled?  The change you described may "appear" to
> be working when in fact it's only working for a subset of real-world
> use cases.

On the MIPS STB, PID filtering is always disabled. The highest bandwidth
modulation I have available in my area is a QAM256 DVB-T2 multiplex, and
it appears to be working fine.

Lowering the buffer size below 4k, I can see some image corruption
(green stripes of macro-blocks). So 4k must be on the edge.

I have not tried to capture the entire stream yet. So far, I'm basically
just using the devices.

> Also, the buffer management may be appropriate for the em2874/em2884,
> but be broken for other devices such as the em288[0123].  Testing
> would be required to ensure it's not introducing regressions.  In
> particular, the 2874/2884 had architecture changes which may result in
> differences in behavior (the register map is significantly different,
> for example).

Of course, I'm not proposing to force all existing drivers to use 4k
buffers instead of 60k. What I want to show is that there is room for
optimization in this area. In fact I like the idea of the kernel module
parameter to set the buffer size: the default behavior is unchanged, but
users can test lower values.

>> Also, I prepared a proof-of-concept patch to mimic the usb-dvb URB
>> management; this means the URBs are allocated when the USB device is
>> probed, and are freed when the device is disconnected (the patch code
>> checks for changes in the requested buffer size, but this can never
>> happen in digital mode).
> 
> I don't have any specific problem with such a change assuming it is
> properly vetted against other devices.

Ok.

>> I've tested the patch (with 4k buffers) with both my em28xx sticks and
>> both work perfectly fine on my PC as well as on my MIPS set-top-box.
>> Analog mode is not tested.
> 
> Again, the big question here is surrounding your testing methodology.
> If you could expand on how you're testing, that would be helpful in
> assessing how well the patch will really work.

Please note that the buffer size and the buffer allocation strategy (at
runtime or at driver initialization) are two completely different
topics. The first can cause regressions, the second should not produce
any functional change (bugs excluded).

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Devin Heitmueller]

On Tue, Feb 7, 2012 at 11:01 AM, Gianluca Gennari <gennarone@gmail.com> wrote:
> Please note that the buffer size and the buffer allocation strategy (at
> runtime or at driver initialization) are two completely different
> topics. The first can cause regressions, the second should not produce
> any functional change (bugs excluded).

Agreed.  I would break this into two patches and submit the buffer
allocation strategy change first.  I think that should be able to go
upstream without too much debate/discussion.  From there we can
discuss the merits of the various approaches for the second patch
(relating to the buffer size).

Cheers,

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

Re: PCTV 290e page allocation failure

[Gianluca Gennari]

Il 07/02/2012 15:57, Andy Furniss ha scritto:

> It will still fail if it has already failed and not been replugged.
> 
> It's not failing to allocate - it's just not trying to allocate AFAICT ,
> which I guess counts as a bug?

For what is worth, on the MIPS STB I can't even rmmod the em28xx module
and reload it, as rmmod gets stuck.
The only solution to get the PCTV working again is a reboot.

Regards,
Gianluca

Re: PCTV 290e page allocation failure

[Devin Heitmueller]

On Tue, Feb 7, 2012 at 1:10 PM, Gianluca Gennari <gennarone@gmail.com> wrote:
> Il 07/02/2012 15:57, Andy Furniss ha scritto:
>
>> It will still fail if it has already failed and not been replugged.
>>
>> It's not failing to allocate - it's just not trying to allocate AFAICT ,
>> which I guess counts as a bug?
>
> For what is worth, on the MIPS STB I can't even rmmod the em28xx module
> and reload it, as rmmod gets stuck.
> The only solution to get the PCTV working again is a reboot.

Which kernel are you running.  There were fixes for problems related
to rmmod em28xx not working, which were only fixed recently.  You
would have to check the linux-media ML archives for the actual details
in terms of what release the work was committed to.

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com