kmalloc/pci_alloc and skbuff's

kmalloc/pci_alloc and skbuff's

[Geoffrey Espin]

The Markham board (Korva/VR41xx variant) which I'm trying to get
submitted, has a PCI problem.  It doesn't allow a PCI device to
see all the DRAM.  DMA-able data must be in a special 4MB range,
which is currently set at 0x01c00000 (the last 4MB of a 32MB
system). SKB's must be allocated within this area for a
(say, Ethernet) PCI device to access. There appears to be no way
to tell kmalloc(), used in alloc_skb(), to allocate memory from
any special address range.

The pci_{alloc,free}_consistent() routines do the right thing.

Hence the following hack to linux/net/core/skbuff.c.

Any suggestions?

Geoff
-- 
Geoffrey Espin espin@idiom.com
--

--- skbuff.c.ORIG	Tue Aug  7 08:30:50 2001
+++ skbuff.c	Mon Dec 17 13:46:40 2001
@@ -62,6 +62,12 @@
 #include <asm/uaccess.h>
 #include <asm/system.h>
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+#warning PCI ALLOC: see linux/arch/mips/markham/pci_fixup.c
+extern void *pci_alloc_consistent(int, int, dma_addr_t *);
+extern void pci_free_consistent(int,int, void *, int);
+#endif
+
 int sysctl_hot_list_len = 128;
 
 static kmem_cache_t *skbuff_head_cache;
@@ -187,6 +193,23 @@
 
 	/* Get the DATA. Size must match skb_add_mtu(). */
 	size = SKB_DATA_ALIGN(size);
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		goto nodata;
@@ -286,6 +309,12 @@
 		if (skb_shinfo(skb)->frag_list)
 			skb_drop_fraglist(skb);
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+		/* NEC Markham: see allock_skb() also */
+		if (1)
+			pci_free_consistent(0,0,skb->head,0);
+		else
+#endif
 		kfree(skb->head);
 	}
 }
@@ -505,6 +534,24 @@
 
 	size = (skb->end - skb->head + expand);
 	size = SKB_DATA_ALIGN(size);
+
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		return -ENOMEM;
@@ -512,7 +559,6 @@
 	/* Copy entire thing */
 	if (skb_copy_bits(skb, -headerlen, data, headerlen+skb->len))
 		BUG();
-
 	/* Offset between the two in bytes */
 	offset = data - skb->head;
 
@@ -627,6 +673,23 @@
 
 	size = SKB_DATA_ALIGN(size);
 
+#if defined(CONFIG_NEC_MARKHAM_PCI)
+	/* NEC Markham: see skb_release_data() also */
+
+	/*
+	 * Markham doesn't allow a PCI device to see all the RAM.
+	 * DMAable data must be on this special 4MB range, which
+	 * is currently set at 0x01c00000. SKB buffers must also
+	 * be allocated within this area for a PCI device to
+	 * access. There is no way to tell kmalloc() to allocate
+	 * memory from any special address range. So we do it on
+	 * our own.
+	 *                     Dai, Wed Jun 13 19:34:23 PDT 2001
+	 */
+	if (1)
+		data = (u8 *)pci_alloc_consistent(0,size + sizeof(struct skb_shared_info), NULL);
+	else
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (data == NULL)
 		goto nodata;

Re: kmalloc/pci_alloc and skbuff's

[Gleb O. Raiko]

Geoffrey Espin wrote:
> 
> The Markham board (Korva/VR41xx variant) which I'm trying to get
> submitted, has a PCI problem.  It doesn't allow a PCI device to
> see all the DRAM.  DMA-able data must be in a special 4MB range,
> which is currently set at 0x01c00000 (the last 4MB of a 32MB
> system). SKB's must be allocated within this area for a
> (say, Ethernet) PCI device to access. There appears to be no way
> to tell kmalloc(), used in alloc_skb(), to allocate memory from
> any special address range.
> 

What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Regards,
Gleb.

Re: kmalloc/pci_alloc and skbuff's

[James Simmons]

> What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?

Personally I don't like the hack but you have to ask what he needs.
kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory
is continues. I think Geoffrey needs to use a specific memory address in 
PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't
like the hack is that skbuff's is bus independent. Not all ethernet cards
are PCI based. Please try using GFP_DMA and let us know if it worked. 

   . ---
   |o_o |
   |:_/ |   Give Micro$oft the Bird!!!!
  //   \ \  Use Linux!!!!
 (|     | )
 /'_   _/`\
 ___)=(___/

Re: kmalloc/pci_alloc and skbuff's

[Geoffrey Espin]

> > What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?
> Personally I don't like the hack but you have to ask what he needs.
> kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory
> is continues. I think Geoffrey needs to use a specific memory address in 
> PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't
> like the hack is that skbuff's is bus independent. Not all ethernet cards
> are PCI based. Please try using GFP_DMA and let us know if it worked. 

Yes, I originally thought this was what addressed it.
Is "setting dma_mask" what is meant by "using GFP_DMA"?

The problem is drivers call dev_alloc_skb() which can allocate
memory anywhere in (my 32M) memory.

The PCI host controller part of the uPD98052 with its VR4120a core
(doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
allows you to program a 4M window onto DRAM.  I use top 4M of 32M,
but it's arbitrary.  Then only this area can be transferred to
by/from the PCI devices.  So its not the PCI devices that is the
problem, but access to the host-side DRAM.

Currently, my private pci_alloc/free_consistent() routines manage
the 4M at top of memory (its not added to kernel with
add_memory_region() in prom.c).

With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent)
I've been successfully using the Tulip Ethernet (LinkSys) card (with no
changes to the driver).

Geoff
-- 
Geoffrey Espin
espin@idiom.com

Re: kmalloc/pci_alloc and skbuff's

[Pete Popov]

On Wed, 2001-12-19 at 10:56, Geoffrey Espin wrote:
> 
> > > What's wrong with GFP_DMA ? Doesn't it solve exactly this problem ?
> > Personally I don't like the hack but you have to ask what he needs.
> > kmalloc grabs memory from the CPU cache. GFP_DMA insures that cache memory
> > is continues. I think Geoffrey needs to use a specific memory address in 
> > PCI space. Tho I like Geoffrey to try using GFP_DMA. The reason I don't
> > like the hack is that skbuff's is bus independent. Not all ethernet cards
> > are PCI based. Please try using GFP_DMA and let us know if it worked. 
> 
> Yes, I originally thought this was what addressed it.
> Is "setting dma_mask" what is meant by "using GFP_DMA"?
> 
> The problem is drivers call dev_alloc_skb() which can allocate
> memory anywhere in (my 32M) memory.
> 
> The PCI host controller part of the uPD98052 with its VR4120a core
> (doc at http://www.idiom.com/~espin/nec/hwdoc/uPD98502-UM.pdf)
> allows you to program a 4M window onto DRAM.  I use top 4M of 32M,
> but it's arbitrary.  Then only this area can be transferred to
> by/from the PCI devices.  So its not the PCI devices that is the
> problem, but access to the host-side DRAM.
> 
> Currently, my private pci_alloc/free_consistent() routines manage
> the 4M at top of memory (its not added to kernel with
> add_memory_region() in prom.c).
> 
> With these hacks (including net/core/skbuff.c:alloc_skb->pci_alloc_consistent)
> I've been successfully using the Tulip Ethernet (LinkSys) card (with no
> changes to the driver).

FYI, this is not an isolated issue. We deal with a number of
architectures and we've seen this problem with other arches and system
controllers as well. A 'generic' solution would be nice and probably
necessary at some point. 2.5 would be a good place to do it, if only
someone would volunteer ;-)

Pete

Re: kmalloc/pci_alloc and skbuff's

[Geoffrey Espin]

> > Currently, my private pci_alloc/free_consistent() routines manage

> FYI, this is not an isolated issue. We deal with a number of
> architectures and we've seen this problem with other arches and system
> controllers as well. A 'generic' solution would be nice and probably
> necessary at some point. 2.5 would be a good place to do it, if only
> someone would volunteer ;-)   [Pete]

Thanks for the reassurance.

Can one include one's own arch/mips/korva/skbuff.c?
But with network.o being a monolithic blob .o instead of a .a,
seems like patching the one file is not feasible.
I tried tweaking $(HEAD) but then stumbled onto this.

How does one package such a work-around?  Include a patch file in
the LSP, which gets automatically run to munge on kernel sources?

Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Geoff
-- 
Geoffrey Espin espin@idiom.com

Re: kmalloc/pci_alloc and skbuff's

[Pete Popov]

On Wed, 2001-12-19 at 11:53, Geoffrey Espin wrote:
> 
> > > Currently, my private pci_alloc/free_consistent() routines manage
> 
> > FYI, this is not an isolated issue. We deal with a number of
> > architectures and we've seen this problem with other arches and system
> > controllers as well. A 'generic' solution would be nice and probably
> > necessary at some point. 2.5 would be a good place to do it, if only
> > someone would volunteer ;-)   [Pete]
> 
> Thanks for the reassurance.
> 
> Can one include one's own arch/mips/korva/skbuff.c?
> But with network.o being a monolithic blob .o instead of a .a,
> seems like patching the one file is not feasible.
> I tried tweaking $(HEAD) but then stumbled onto this.
 
> How does one package such a work-around?  Include a patch file in
> the LSP, which gets automatically run to munge on kernel sources?

If you mean MontaVista's LSP, they are built from an internal source
tree, so in some sense we can do whatever we want -- at least until we
implement the right solution. It's not something you want to do though
because carrying patches with you does not scale well.   The best
approach in the long run is to do it right and push it out to the
community tree asap.  The problem, of course, is that if you yourself
wanted to do it right, but it was going to take a while, the company
you're consulting for might not want to pay for such general linux work.
 
> Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?

Up the the maintainer, but it's really not the right fix so I think it's
best for you or your customer to keep that patch locally.

Another approach would be for you to preallocate your network buffers in
your driver, attach them permanently to the rx/tx descriptors, and then
copy the data from the buffer to a newly allocated skbuff, when you
receive, and the other way when you transmit.  There's a performance
penalty there, especially for large packets, but you can encapsulate
everything within your driver and you won't have to maintain a patch.

Pete

Re: kmalloc/pci_alloc and skbuff's

[Geoffrey Espin]

> > Can one include one's own arch/mips/korva/skbuff.c?
> > But with network.o being a monolithic blob .o instead of a .a,

I figured someone might have some magic for hacking
arch/mips/Makefile ifdef CONFIG_NEC_KORVA (or any BSP/LSP).
I guess I'll have to make the effort.  :-)

> > Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?
> Up the the maintainer, but it's really not the right fix so I think it's
> best for you or your customer to keep that patch locally.

I talked Jun into checking in the Markham patches... but will
have to add a comment to the pci_xxx files that one has to get
a separate patch for net/core/skbuff.c if using PCI with a generic
driver.

> Another approach would be for you to preallocate your network buffers in
> your driver, attach them permanently to the rx/tx descriptors, and then

The idea was not to mess with any PCI drivers. 2.5 is not an option for me.

Thanks, Pete.

Geoff
-- 
Geoffrey Espin espin@idiom.com

Re: kmalloc/pci_alloc and skbuff's

[Pete Popov]

On Wed, 2001-12-19 at 13:56, Geoffrey Espin wrote:
> > > Can one include one's own arch/mips/korva/skbuff.c?
> > > But with network.o being a monolithic blob .o instead of a .a,
> 
> I figured someone might have some magic for hacking
> arch/mips/Makefile ifdef CONFIG_NEC_KORVA (or any BSP/LSP).
> I guess I'll have to make the effort.  :-)
> 
> > > Or will linux-mips.sf.net accept a patch for net/core/skbuff.c?
> > Up the the maintainer, but it's really not the right fix so I think it's
> > best for you or your customer to keep that patch locally.
> 
> I talked Jun into checking in the Markham patches... but will
> have to add a comment to the pci_xxx files that one has to get
> a separate patch for net/core/skbuff.c if using PCI with a generic
> driver.
> 
> > Another approach would be for you to preallocate your network buffers in
> > your driver, attach them permanently to the rx/tx descriptors, and then
> 
> The idea was not to mess with any PCI drivers. 2.5 is not an option for me.

Sorry, that was a bad suggestion.  I was thinking of an ethernet driver
that's part of your SOC and requires a new ethernet driver anyway. 
Otherwise, messing with all the pci ethernet drivers is definitely not
an option.

Pete