Re: Memory problem with bttv driver

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

> On Mon, 14 Jan 2002, Stephan von Krawczynski wrote:                 
> >                                                                   
> > Ok. So what do we do about it? I mean there are possibly some more
people out                                                            
> > there with such a problem, or - to my prediction - there will be  
more in the                                                           
> > future. I see to possibilities:                                   
> > 1) simply increase it overall. I have not the slightest idea what 
the drawbacks                                                         
> > are. 2) make it configurable (looks like general setup to me).    
> >                                                                   
> > I could provide a patch for either. Do we want that?              
>                                                                     
> I was waiting for hpa to jump on you for suggesting that!  but since
> he hasn't yet done so (publicly), here's his mail when I suggested  
it                                                                    
> a month ago (as a temporary aid while tracking down a similar       
problem).                                                             
>                                                                     
> That problem was from NTFS overusing vmalloc (and I think Anton has 
> now posted the fix), but NTFS wasn't in your list, and I assume you 
> didn't have it in statically.                                       
                                                                      
Nope, no ntfs involved here.                                          
I read the former thread. It was in my mind when trying the increased 
VMALLOC_RESERVE, and voila, it worked again. I must admit in your case
I was very much thinking that ntfs should be fixed, and I am pleased  
the maintainer thought the same :-)                                   
Unfortunately I do not have a good idea about what to do in days where
the graphics cards have more ram onboard than my last workstation had 
in total. If I understood Alans' opinion, he thinks basically the     
same, what the heck can you do about such monster cards? I don't know,
other than adjusting the RESERVE-area. I have not really thought about
it right now, but I suspect it would be rather impossible to make a   
somehow runtime-scalable area working, which would of course be the   
complete solution to the underlying problem.                          
                                                                      
So, what's left? At least configurable? Anybody with a better idea?   
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
>                                                                     

Memory problem with bttv driver

[Stephan von Krawczynski]

Hello,

yesterday we came across another MM related problem. This time its related to
the bttv-driver. We do very simple things:

First load it:

# modprobe bttv
Jan 14 17:28:14 fred kernel: bttv: driver version 0.7.83 loaded
Jan 14 17:28:14 fred kernel: bttv: using 2 buffers with 2080k (4160k total) for
capture Jan 14 17:28:14 fred kernel: bttv: Host bridge is VIA Technologies,
Inc. VT82C693A/694x [Apollo PRO133x] Jan 14 17:28:14 fred kernel: bttv: Host
bridge is VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] Jan 14 17:28:14
fred kernel: bttv: Bt8xx card found (0). Jan 14 17:28:14 fred kernel: bttv0:
Bt848 (rev 18) at 00:0c.0, irq: 16, latency: 32, memory: 0xef000000 Jan 14
17:28:14 fred kernel: bttv0: using: BT848A( *** UNKNOWN/GENERIC **)
[card=0,autodetected] Jan 14 17:28:14 fred kernel: i2c-core.o: adapter bt848 #0
registered as adapter 0. Jan 14 17:28:14 fred kernel: bttv0: i2c: checking for
MSP34xx @ 0x80... not found Jan 14 17:28:14 fred kernel: bttv0: i2c: checking
for TDA9875 @ 0xb0... not found Jan 14 17:28:14 fred kernel: bttv0: i2c:
checking for TDA7432 @ 0x8a... not found Jan 14 17:28:14 fred kernel:
i2c-core.o: driver i2c TV tuner driver registered. Jan 14 17:28:14 fred kernel:
tuner: chip found @ 0xc2 Jan 14 17:28:14 fred kernel: bttv0: i2c attach
[client=(unset),ok] Jan 14 17:28:14 fred kernel: i2c-core.o: client [(unset)]
registered to adapter [bt848 #0](pos. 0).

Then try to use it:

# xawtv
Jan 14 17:46:53 fred kernel: bttv: vmalloc_32(4259840) failed

... and exit of course.

Uh?
Interesting about it:
Does not work with 2.4.17 nor 2.4.18-pre3, but does work with 2.4.10-SUSE.
(Same setup, only booting another kernel)

Any hints?

Regards,
Stephan

Re: Memory problem with bttv driver

[Alan Cox]

> yesterday we came across another MM related problem. This time its related to
> the bttv-driver. We do very simple things:

Do you trust your kernel build ? The bt848 drivers are working beautifully
for me in 2.4.18pre

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 17:40:24 +0000 (GMT)
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > yesterday we came across another MM related problem. This time its related
to> > the bttv-driver. We do very simple things:
> 
> Do you trust your kernel build ?

Yes.

> The bt848 drivers are working beautifully
> for me in 2.4.18pre

Well, I had a quick look at the code, and it seems that vmalloc is just
failing, the source line is obvious./proc/meminfo before modprobe and xawtv:

        total:    used:    free:  shared: buffers:  cached:
Mem:  1054728192 120070144 934658048        0 10420224 65257472
Swap: 1085652992        0 1085652992
MemTotal:      1030008 kB
MemFree:        912752 kB
MemShared:           0 kB
Buffers:         10176 kB
Cached:          63728 kB
SwapCached:          0 kB
Active:          26056 kB
Inactive:        73692 kB
HighTotal:      131056 kB
HighFree:        36720 kB
LowTotal:       898952 kB
LowFree:        876032 kB
SwapTotal:     1060208 kB
SwapFree:      1060208 kB

And afterwards:

        total:    used:    free:  shared: buffers:  cached:
Mem:  1054728192 120537088 934191104        0 10612736 65273856
Swap: 1085652992        0 1085652992
MemTotal:      1030008 kB
MemFree:        912296 kB
MemShared:           0 kB
Buffers:         10364 kB
Cached:          63744 kB
SwapCached:          0 kB
Active:          26140 kB
Inactive:        73812 kB
HighTotal:      131056 kB
HighFree:        36608 kB
LowTotal:       898952 kB
LowFree:        875688 kB
SwapTotal:     1060208 kB
SwapFree:      1060208 kB

Can this be highmem-related?

Regards,
Stephan

Re: Memory problem with bttv driver

[Alan Cox]

> > The bt848 drivers are working beautifully
> > for me in 2.4.18pre
> 
> Well, I had a quick look at the code, and it seems that vmalloc is just
> failing, the source line is obvious./proc/meminfo before modprobe and xawtv:
> 
>         total:    used:    free:  shared: buffers:  cached:
> Mem:  1054728192 120070144 934658048        0 10420224 65257472
> Swap: 1085652992        0 1085652992
> 
> Can this be highmem-related?

That would make complete sense if so. The bttv uses vmalloc_32(), as the
card has 32bit limits, and I am not running bttv (nor I suspect are most
people) with highmem enabled

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 17:56:54 +0000 (GMT)
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > > The bt848 drivers are working beautifully
> > > for me in 2.4.18pre
> > 
> > Well, I had a quick look at the code, and it seems that vmalloc is just
> > failing, the source line is obvious./proc/meminfo before modprobe and
xawtv:> > 
> >         total:    used:    free:  shared: buffers:  cached:
> > Mem:  1054728192 120070144 934658048        0 10420224 65257472
> > Swap: 1085652992        0 1085652992
> > 
> > Can this be highmem-related?
> 
> That would make complete sense if so. The bttv uses vmalloc_32(), as the
> card has 32bit limits, and I am not running bttv (nor I suspect are most
> people) with highmem enabled

Ok, we re-checked without highmem: it's still the same problem. I try to find
out what's so special about 2.4.10-SUSE...

Sorry for this dumb newbie question: is there an easy way (/proc?) to find out
how much vmalloc space is used/left?

Regards,
Stephan

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 17:56:54 +0000 (GMT)
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > Can this be highmem-related?
> 
> That would make complete sense if so. The bttv uses vmalloc_32(), as the
> card has 32bit limits, and I am not running bttv (nor I suspect are most
> people) with highmem enabled

Ok. I tracked it down. It is definitely VMALLOC-stuff. I increased the
VMALLOC_RESERVE from 128 to 256 MB and now it _works_. Here is a list of loaded
modules, maybe one (or several) of those are known to be vmalloc-fans :-)


Module                  Size  Used by
tuner                   8048   1  (autoclean)
bttv                   60848   0 
i2c-algo-bit            7040   1  [bttv]
i2c-core               12224   0  [tuner bttv i2c-algo-bit]
videodev                4768   2  [bttv]
NVdriver              720128  14  (autoclean)
parport_pc             12432   1  (autoclean)
lp                      5984   0  (autoclean)
parport                12736   1  (autoclean) [parport_pc lp]
nfs                    73024   2  (autoclean)
lockd                  47056   1  (autoclean) [nfs]
sunrpc                 62496   1  (autoclean) [nfs lockd]
ipv6                  158464  -1  (autoclean)
uhci                   24896   0  (unused)
usbcore                48640   1  [uhci]
3c59x                  25312   1  (autoclean)
emu10k1                58080   0 
sound                  54064   0  [emu10k1]
ac97_codec              9504   0  [emu10k1]
hisax                 168112   4 
isdn                  115088   6  [hisax]
slhc                    4304   0  [isdn]
serial                 44128   0  (autoclean)


How can I find out?

Regards,
Stephan

Re: Memory problem with bttv driver

[Alan Cox]

> bttv                   60848   0 

bttv wants a reasonable amount

> NVdriver              720128  14  (autoclean)

Thats my guess. Mapping all the memory on your geofarce will not be a small
amount.

Alan

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 20:03:49 +0000 (GMT)
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > bttv                   60848   0 
> 
> bttv wants a reasonable amount
> 
> > NVdriver              720128  14  (autoclean)
> 
> Thats my guess. Mapping all the memory on your geofarce will not be a small
> amount.

Ok. So what do we do about it? I mean there are possibly some more people out
there with such a problem, or - to my prediction - there will be more in the
future. I see to possibilities:
1) simply increase it overall. I have not the slightest idea what the drawbacks
are. 2) make it configurable (looks like general setup to me).

I could provide a patch for either. Do we want that?

Regards,
Stephan

Re: Memory problem with bttv driver

[Alan Cox]

> Ok. So what do we do about it? I mean there are possibly some more people out
> there with such a problem, or - to my prediction - there will be more in the
> future. I see to possibilities:
> 1) simply increase it overall. I have not the slightest idea what the drawbacks
> are. 2) make it configurable (looks like general setup to me).

Making it bigger reduces that amount of ram directly mapped by the kernel 
which hits performance (nastily for 2.4 not so bad for 2.5)

Re: Memory problem with bttv driver

[J.A. Magallon]

On 20020114 Alan Cox wrote:
>> Ok. So what do we do about it? I mean there are possibly some more people out
>> there with such a problem, or - to my prediction - there will be more in the
>> future. I see to possibilities:
>> 1) simply increase it overall. I have not the slightest idea what the drawbacks
>> are. 2) make it configurable (looks like general setup to me).
>
>Making it bigger reduces that amount of ram directly mapped by the kernel 
>which hits performance (nastily for 2.4 not so bad for 2.5)

(Sorry for joning so late to the thread...)
It is wokring fine for me, under 2.4.18-pre3 + NVidia. The difference is
that I am using version 0.7.87 (see http://giga.cps.unizar.es/~magallon/linux/2.4.18-pre3/)
I have just checked the bttv page (http://bytesex.org/bttv/) and there is
a newer version (0.7.88). What comes in .17 is 0.7.83. I have not
noticed anything relevant in changelog, but you can try...

lsmod:
werewolf:~# lsmod
Module                  Size  Used by    Tainted: P  
tuner                   8548   1  (autoclean)
tvaudio                 9312   0  (autoclean) (unused)
msp3400                14768   1  (autoclean)
bttv                   63424   0  (autoclean)
i2c-algo-bit            7244   1  (autoclean) [bttv]
videodev                4960   3  (autoclean) [bttv]
emu10k1                57728   1 
sound                  55052   0  [emu10k1]
ac97_codec              9472   0  [emu10k1]
soundcore               3524   7  [emu10k1 sound]
serial                 45792   7 
isa-pnp                28968   0  [serial]
w83781d                17792   0  (unused)
i2c-proc                6112   0  [w83781d]
i2c-isa                 1188   0  (unused)
i2c-piix4               3908   0  (unused)
i2c-core               13408   0  [tuner tvaudio msp3400 bttv i2c-algo-bit w83781d i2c-proc i2c-isa i2c-piix4]
NVdriver              821600  14 
agpgart                17024   3 
ne2k-pci                4960   1 
8390                    6608   0  [ne2k-pci]

--
J.A. Magallon                           #  Let the source be with you...        
mailto:jamagallon@able.es
Mandrake Linux release 8.2 (Cooker) for i586
Linux werewolf 2.4.18-pre3-beo #5 SMP Sun Jan 13 02:14:04 CET 2002 i686

Re: Memory problem with bttv driver

[Gerd Knorr]

> >Making it bigger reduces that amount of ram directly mapped by the kernel 
> >which hits performance (nastily for 2.4 not so bad for 2.5)
>  
>  (Sorry for joning so late to the thread...)
>  It is wokring fine for me, under 2.4.18-pre3 + NVidia. The difference is
>  that I am using version 0.7.87 (see http://giga.cps.unizar.es/~magallon/linux/2.4.18-pre3/)
>  I have just checked the bttv page (http://bytesex.org/bttv/) and there is
>  a newer version (0.7.88). What comes in .17 is 0.7.83. I have not
>  noticed anything relevant in changelog, but you can try...

MM wise it shouldn't make a difference whenever you are using 0.7.83 or
0.7.88 (I've mailed 0.7.88 patches to macelo for 2.4.18 btw).  The 0.8.x
versions have a complete different way to do the memory management.

  Gerd

-- 
#define	ENOCLUE 125 /* userland programmer induced race condition */

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On 15 Jan 2002 10:17:03 GMT
Gerd Knorr <kraxel@bytesex.org> wrote:


> MM wise it shouldn't make a difference whenever you are using 0.7.83 or
> 0.7.88 (I've mailed 0.7.88 patches to macelo for 2.4.18 btw).  The 0.8.x
> versions have a complete different way to do the memory management.

No vmallocs?

Regards,
Stephan

Re: Memory problem with bttv driver

[Gerd Knorr]

On Tue, Jan 15, 2002 at 12:14:24PM +0100, Stephan von Krawczynski wrote:
> On 15 Jan 2002 10:17:03 GMT
> Gerd Knorr <kraxel@bytesex.org> wrote:
> 
> > MM wise it shouldn't make a difference whenever you are using 0.7.83 or
> > 0.7.88 (I've mailed 0.7.88 patches to macelo for 2.4.18 btw).  The 0.8.x
> > versions have a complete different way to do the memory management.
> 
> No vmallocs?

Yes.  Instead of remapping vmalloced kernel memory it gives you shared
anonymous pages, then does zerocopy DMA using kiobufs.  You may run in
trouble with >4GB machines.

  Gerd

-- 
#define	ENOCLUE 125 /* userland programmer induced race condition */

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Tue, 15 Jan 2002 14:20:17 +0100
Gerd Knorr <kraxel@bytesex.org> wrote:

> On Tue, Jan 15, 2002 at 12:14:24PM +0100, Stephan von Krawczynski wrote:
> > On 15 Jan 2002 10:17:03 GMT
> > Gerd Knorr <kraxel@bytesex.org> wrote:
> > 
> > > MM wise it shouldn't make a difference whenever you are using 0.7.83 or
> > > 0.7.88 (I've mailed 0.7.88 patches to macelo for 2.4.18 btw).  The 0.8.x
> > > versions have a complete different way to do the memory management.
> > 
> > No vmallocs?
> 
> Yes.  Instead of remapping vmalloced kernel memory it gives you shared
> anonymous pages, then does zerocopy DMA using kiobufs.  You may run in
> trouble with >4GB machines.

Interesting.
What's the problem on > 4GB ?

Regards,
Stephan

Re: Memory problem with bttv driver

[Gerd Knorr]

On Tue, Jan 15, 2002 at 02:46:52PM +0100, Stephan von Krawczynski wrote:
> On Tue, 15 Jan 2002 14:20:17 +0100
> Gerd Knorr <kraxel@bytesex.org> wrote:
> 
> > Yes.  Instead of remapping vmalloced kernel memory it gives you shared
> > anonymous pages, then does zerocopy DMA using kiobufs.  You may run in
> > trouble with >4GB machines.
> 
> Interesting.
> What's the problem on > 4GB ?

The bt878/848 is a 32bit PCI device, it simply can't go DMA to main
memory above 4GB.  At least on ia32, on architecures with a iommu
(sparc, ...) it should work without trouble.

  Gerd

-- 
#define	ENOCLUE 125 /* userland programmer induced race condition */

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Tue, 15 Jan 2002 16:16:53 +0100
Gerd Knorr <kraxel@bytesex.org> wrote:

> On Tue, Jan 15, 2002 at 02:46:52PM +0100, Stephan von Krawczynski wrote:
> > On Tue, 15 Jan 2002 14:20:17 +0100
> > Gerd Knorr <kraxel@bytesex.org> wrote:
> > 
> > > Yes.  Instead of remapping vmalloced kernel memory it gives you shared
> > > anonymous pages, then does zerocopy DMA using kiobufs.  You may run in
> > > trouble with >4GB machines.
> > 
> > Interesting.
> > What's the problem on > 4GB ?
> 
> The bt878/848 is a 32bit PCI device, it simply can't go DMA to main
> memory above 4GB.  At least on ia32, on architecures with a iommu
> (sparc, ...) it should work without trouble.

Sorry, maybe I should have clarified: what is the problem with allocating those
pages below 4GB in a >4GB box?

Regards,
Stephan

Re: Memory problem with bttv driver

[Gerd Knorr]

> > > > Yes.  Instead of remapping vmalloced kernel memory it gives you shared
> > > > anonymous pages, then does zerocopy DMA using kiobufs.  You may run in
> > > > trouble with >4GB machines.
> > > 
> > > Interesting.
> > > What's the problem on > 4GB ?
> > 
> > The bt878/848 is a 32bit PCI device, it simply can't go DMA to main
> > memory above 4GB.  At least on ia32, on architecures with a iommu
> > (sparc, ...) it should work without trouble.
> 
> Sorry, maybe I should have clarified: what is the problem with allocating those
> pages below 4GB in a >4GB box?

do_anonymous_page() may give you pages above 4GB.  Need to write my own
nopage handler to fix this.

  Gerd

-- 
#define	ENOCLUE 125 /* userland programmer induced race condition */

Re: Memory problem with bttv driver

[Stephan von Krawczynski]

On Wed, 16 Jan 2002 10:57:47 +0100
Gerd Knorr <kraxel@bytesex.org> wrote:

> > > > > Yes.  Instead of remapping vmalloced kernel memory it gives you
shared> > > > > anonymous pages, then does zerocopy DMA using kiobufs.  You may
run in> > > > > trouble with >4GB machines.
> > > > 
> > > > Interesting.
> > > > What's the problem on > 4GB ?
> > > 
> > > The bt878/848 is a 32bit PCI device, it simply can't go DMA to main
> > > memory above 4GB.  At least on ia32, on architecures with a iommu
> > > (sparc, ...) it should work without trouble.
> > 
> > Sorry, maybe I should have clarified: what is the problem with allocating
those> > pages below 4GB in a >4GB box?
> 
> do_anonymous_page() may give you pages above 4GB.

Guess it is a bit ... flaky ... to just do a

                page = alloc_page(GFP_HIGHUSER);
                if (!page)
                        goto no_mem;

in

do_anonymous_page()

I guess it should be somehow configurable (parameter). Your idea of using it
may grow and mem too, for sure.

Regards,
Stephan