Socket buffer allocation outside DMA-able memory

Socket buffer allocation outside DMA-able memory

[art]

Hello all!
I work with ADM5120 chip. it has embedded switch.
Switch descriptor has 25-bit dma addres field - so addressible only
32Mb!
My system has 64Mb memory. But I have to set 32Mb to make it work!
I thought that setting DMA mask can help. So in
/arch/mips/adm5120/setup.c i make:

static struct platform_device adm5120hcd_device = {
        .name           = "adm5120-hcd",
        .id             = -1,
        .dev            = {
        .dma_mask               = &hcd_dmamask,
        .coherent_dma_mask      = 0x01ffffff,
        },
        .num_resources  = ARRAY_SIZE(adm5120_hcd_resources),
        .resource       = adm5120_hcd_resources,
};
But It is wrong, because dev_alloc_skb dont know to which device it
allocates buffer!

How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
system has 64Mb?

-- 
Best regards,
 art                          mailto:art@sigrand.ru

Re: Socket buffer allocation outside DMA-able memory

[ashley jones]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

hi,
         I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
   
          If that is not the case then you can use one of the foll. work around,
   
    * dont give whole 64 MB to linux, give only Lower 32 MB.
    * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit in this case.)
   
   
  Regards,
  A'Jones.
  

art <art@sigrand.ru> wrote:
  Hello all!
I work with ADM5120 chip. it has embedded switch.
Switch descriptor has 25-bit dma addres field - so addressible only
32Mb!
My system has 64Mb memory. But I have to set 32Mb to make it work!
I thought that setting DMA mask can help. So in
/arch/mips/adm5120/setup.c i make:

static struct platform_device adm5120hcd_device = {
.name = "adm5120-hcd",
.id = -1,
.dev = {
.dma_mask = &hcd_dmamask,
.coherent_dma_mask = 0x01ffffff,
},
.num_resources = ARRAY_SIZE(adm5120_hcd_resources),
.resource = adm5120_hcd_resources,
};
But It is wrong, because dev_alloc_skb dont know to which device it
allocates buffer!

How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
system has 64Mb?

-- 
Best regards,
art mailto:art@sigrand.ru





__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[-- Attachment #2: Type: text/html, Size: 1940 bytes --]

Re[2]: Socket buffer allocation outside DMA-able memory

[art]

Hello ashley,

Tuesday, June 06, 2006, 7:37:35 PM, you wrote:


aj>          I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
Address not aligned - if I don't do anything driver work incorrect!

aj>     * dont give whole 64 MB to linux, give only Lower 32 MB.
You mean with command line kernel option? If so - I already did so to
get all work!

aj>     * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit
aj> in this case.)
How can I do this? I have variant that if addres is upper than 32Mb then copy skbuffer to
previously allocated memory that lower than 32Mb, but perfomance is wery Important. Maybe
you mean this??

Re: Re[2]: Socket buffer allocation outside DMA-able memory

[ashley jones]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]


art <art@sigrand.ru> wrote:    Hello ashley,

Tuesday, June 06, 2006, 7:37:35 PM, you wrote:


aj> I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
Address not aligned - if I don't do anything driver work incorrect!
   
  *** what do you mean by Address not aligned ??
   
  *** What address you r passing to dma ? u should pass (skb->data >> 2) (if word alligned address is required for dma engine.)

aj> * dont give whole 64 MB to linux, give only Lower 32 MB.
You mean with command line kernel option? If so - I already did so to
get all work!

  *** when you do add_memory_region(), give only first 32 MB to linux, so all the skbs will be in lower 32 MB range.
   
  aj> * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit
aj> in this case.)
How can I do this? I have variant that if addres is upper than 32Mb then copy skbuffer to
previously allocated memory that lower than 32Mb, but perfomance is wery Important. Maybe
you mean this??




A'Jones.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[-- Attachment #2: Type: text/html, Size: 1659 bytes --]

Re[4]: Socket buffer allocation outside DMA-able memory

[art]

Hello ashley,

Wednesday, June 07, 2006, 12:17:56 PM, you wrote:



aj>> I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
aj> Address not aligned - if I don't do anything driver work incorrect!
   
aj>   *** what do you mean by Address not aligned ??
   
aj>   *** What address you r passing to dma ? u should pass (skb->data >> 2) (if word alligned address is required for dma engine.)

In adm5120 switch driver to dma passed skb->data

-- 
Best regards,
 art                            mailto:art@sigrand.ru

RE: Socket buffer allocation outside DMA-able memory

[Zhan, Rongkai]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

Hi,

 

Maybe you can enable ISA bus. And then add your low 32MB memory into ZONE_DMA, while the high 32MB memory into ZONE_NORMAL.

In the case, you are required to redefine MAX_DMA_ADDRESS to (PAGE_OFFSET + 0x00200000) in asm-mips/dma.h

 

 

Just a noise.

 

Best Regards,
Mark. Zhan

________________________________

From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of ashley jones
Sent: Tuesday, June 06, 2006 8:38 PM
To: art; linux-mips@linux-mips.org
Subject: Re: Socket buffer allocation outside DMA-able memory

 

hi,

       I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).

 

        If that is not the case then you can use one of the foll. work around,

 

  * dont give whole 64 MB to linux, give only Lower 32 MB.

  * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit in this case.)

 

 

Regards,

A'Jones.



art <art@sigrand.ru> wrote:

	Hello all!
	I work with ADM5120 chip. it has embedded switch.
	Switch descriptor has 25-bit dma addres field - so addressible only
	32Mb!
	My system has 64Mb memory. But I have to set 32Mb to make it work!
	I thought that setting DMA mask can help. So in
	/arch/mips/adm5120/setup.c i make:
	
	static struct platform_device adm5120hcd_device = {
	.name = "adm5120-hcd",
	.id = -1,
	.dev = {
	.dma_mask = &hcd_dmamask,
	.coherent_dma_mask = 0x01ffffff,
	},
	.num_resources = ARRAY_SIZE(adm5120_hcd_resources),
	.resource = adm5120_hcd_resources,
	};
	But It is wrong, because dev_alloc_skb dont know to which device it
	allocates buffer!
	
	How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
	system has 64Mb?
	
	-- 
	Best regards,
	art mailto:art@sigrand.ru
	
	
	

 

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


[-- Attachment #2: Type: text/html, Size: 9082 bytes --]

RE: Socket buffer allocation outside DMA-able memory

[Zhan, Rongkai]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

Hi,

 

Maybe you can enable ISA bus. And then add your low 32MB memory into ZONE_DMA, while the high 32MB memory into ZONE_NORMAL.

In the case, you are required to redefine MAX_DMA_ADDRESS to (PAGE_OFFSET + 0x00200000) in asm-mips/dma.h

 

 

Just a noise.

 

Best Regards,
Mark. Zhan

________________________________

From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of ashley jones
Sent: Tuesday, June 06, 2006 8:38 PM
To: art; linux-mips@linux-mips.org
Subject: Re: Socket buffer allocation outside DMA-able memory

 

hi,

       I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).

 

        If that is not the case then you can use one of the foll. work around,

 

  * dont give whole 64 MB to linux, give only Lower 32 MB.

  * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit in this case.)

 

 

Regards,

A'Jones.



art <art@sigrand.ru> wrote:

	Hello all!
	I work with ADM5120 chip. it has embedded switch.
	Switch descriptor has 25-bit dma addres field - so addressible only
	32Mb!
	My system has 64Mb memory. But I have to set 32Mb to make it work!
	I thought that setting DMA mask can help. So in
	/arch/mips/adm5120/setup.c i make:
	
	static struct platform_device adm5120hcd_device = {
	.name = "adm5120-hcd",
	.id = -1,
	.dev = {
	.dma_mask = &hcd_dmamask,
	.coherent_dma_mask = 0x01ffffff,
	},
	.num_resources = ARRAY_SIZE(adm5120_hcd_resources),
	.resource = adm5120_hcd_resources,
	};
	But It is wrong, because dev_alloc_skb dont know to which device it
	allocates buffer!
	
	How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
	system has 64Mb?
	
	-- 
	Best regards,
	art mailto:art@sigrand.ru
	
	
	

 

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


[-- Attachment #2: Type: text/html, Size: 9082 bytes --]

RE: Socket buffer allocation outside DMA-able memory

[Zhan, Rongkai]

After having a look at the latest 2.6.17-rc6 codes, __dev_alloc_skb is defined like this:

#ifndef CONFIG_HAVE_ARCH_DEV_ALLOC_SKB
/**
 *	__dev_alloc_skb - allocate an skbuff for sending
 *	@length: length to allocate
 *	@gfp_mask: get_free_pages mask, passed to alloc_skb
 *
 *	Allocate a new &sk_buff and assign it a usage count of one. The
 *	buffer has unspecified headroom built in. Users should allocate
 *	the headroom they think they need without accounting for the
 *	built in space. The built in space is used for optimisations.
 *
 *	%NULL is returned in there is no free memory.
 */
static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
					      gfp_t gfp_mask)
{
	struct sk_buff *skb = alloc_skb(length + NET_SKB_PAD, gfp_mask);
	if (likely(skb))
		skb_reserve(skb, NET_SKB_PAD);
	return skb;
}
#else
extern struct sk_buff *__dev_alloc_skb(unsigned int length, int gfp_mask);
#endif


Therefore, you also can consider to implement your machine-specific __dev_alloc_skb() function, and force the skb is allocated from your low 32MB memory zone.

Best Regards,
Mark. Zhan
________________________________________
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Zhan, Rongkai
Sent: Tuesday, June 06, 2006 9:34 PM
To: ashley jones; art; linux-mips@linux-mips.org
Subject: RE: Socket buffer allocation outside DMA-able memory

Hi,

Maybe you can enable ISA bus. And then add your low 32MB memory into ZONE_DMA, while the high 32MB memory into ZONE_NORMAL.
In the case, you are required to redefine MAX_DMA_ADDRESS to (PAGE_OFFSET + 0x00200000) in asm-mips/dma.h


Just a noise.

Best Regards,
Mark. Zhan
________________________________________
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of ashley jones
Sent: Tuesday, June 06, 2006 8:38 PM
To: art; linux-mips@linux-mips.org
Subject: Re: Socket buffer allocation outside DMA-able memory

hi,
       I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
 
        If that is not the case then you can use one of the foll. work around,
 
  * dont give whole 64 MB to linux, give only Lower 32 MB.
  * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit in this case.)
 
 
Regards,
A'Jones.


art <art@sigrand.ru> wrote:
Hello all!
I work with ADM5120 chip. it has embedded switch.
Switch descriptor has 25-bit dma addres field - so addressible only
32Mb!
My system has 64Mb memory. But I have to set 32Mb to make it work!
I thought that setting DMA mask can help. So in
/arch/mips/adm5120/setup.c i make:

static struct platform_device adm5120hcd_device = {
.name = "adm5120-hcd",
.id = -1,
.dev = {
.dma_mask = &hcd_dmamask,
.coherent_dma_mask = 0x01ffffff,
},
.num_resources = ARRAY_SIZE(adm5120_hcd_resources),
.resource = adm5120_hcd_resources,
};
But It is wrong, because dev_alloc_skb dont know to which device it
allocates buffer!

How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
system has 64Mb?

-- 
Best regards,
art mailto:art@sigrand.ru


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: Socket buffer allocation outside DMA-able memory

[Zhan, Rongkai]

After having a look at the latest 2.6.17-rc6 codes, __dev_alloc_skb is defined like this:

#ifndef CONFIG_HAVE_ARCH_DEV_ALLOC_SKB
/**
 *	__dev_alloc_skb - allocate an skbuff for sending
 *	@length: length to allocate
 *	@gfp_mask: get_free_pages mask, passed to alloc_skb
 *
 *	Allocate a new &sk_buff and assign it a usage count of one. The
 *	buffer has unspecified headroom built in. Users should allocate
 *	the headroom they think they need without accounting for the
 *	built in space. The built in space is used for optimisations.
 *
 *	%NULL is returned in there is no free memory.
 */
static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
					      gfp_t gfp_mask)
{
	struct sk_buff *skb = alloc_skb(length + NET_SKB_PAD, gfp_mask);
	if (likely(skb))
		skb_reserve(skb, NET_SKB_PAD);
	return skb;
}
#else
extern struct sk_buff *__dev_alloc_skb(unsigned int length, int gfp_mask);
#endif


Therefore, you also can consider to implement your machine-specific __dev_alloc_skb() function, and force the skb is allocated from your low 32MB memory zone.

Best Regards,
Mark. Zhan
________________________________________
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Zhan, Rongkai
Sent: Tuesday, June 06, 2006 9:34 PM
To: ashley jones; art; linux-mips@linux-mips.org
Subject: RE: Socket buffer allocation outside DMA-able memory

Hi,

Maybe you can enable ISA bus. And then add your low 32MB memory into ZONE_DMA, while the high 32MB memory into ZONE_NORMAL.
In the case, you are required to redefine MAX_DMA_ADDRESS to (PAGE_OFFSET + 0x00200000) in asm-mips/dma.h


Just a noise.

Best Regards,
Mark. Zhan
________________________________________
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of ashley jones
Sent: Tuesday, June 06, 2006 8:38 PM
To: art; linux-mips@linux-mips.org
Subject: Re: Socket buffer allocation outside DMA-able memory

hi,
       I guess your 25 bit dma address field will be word alligned, so ur dma engine will be able to index up to 64 MB( 25+2 = 27 bits).
 
        If that is not the case then you can use one of the foll. work around,
 
  * dont give whole 64 MB to linux, give only Lower 32 MB.
  * Give only upper 32 MB to linux, and give memory to ur dma engine from lower 32 MB, and once you recv any data you copy to skb and submit to linux. ( ofcourse your performance will get hit in this case.)
 
 
Regards,
A'Jones.


art <art@sigrand.ru> wrote:
Hello all!
I work with ADM5120 chip. it has embedded switch.
Switch descriptor has 25-bit dma addres field - so addressible only
32Mb!
My system has 64Mb memory. But I have to set 32Mb to make it work!
I thought that setting DMA mask can help. So in
/arch/mips/adm5120/setup.c i make:

static struct platform_device adm5120hcd_device = {
.name = "adm5120-hcd",
.id = -1,
.dev = {
.dma_mask = &hcd_dmamask,
.coherent_dma_mask = 0x01ffffff,
},
.num_resources = ARRAY_SIZE(adm5120_hcd_resources),
.resource = adm5120_hcd_resources,
};
But It is wrong, because dev_alloc_skb dont know to which device it
allocates buffer!

How to tell kernel allocate skbuffers in less then 32Mb addrspace whet
system has 64Mb?

-- 
Best regards,
art mailto:art@sigrand.ru


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Socket buffer allocation outside DMA-able memory

[Mark.Zhan]

Hi art,

I think maybe my first noise is more easy to go.

As you know, sk_buff is made of a skb header and the packet data 
container. The skb header is allocated from the slab cache, while the 
packet data container is allocated via kmalloc().

So if you can add your low 32MB memory into ZONE_DMA, then your can call 
__dev_alloc_skb() with __GFP_DMA flag in your driver, which should make 
sure that the packet data container is in ZONE_DMA.

Best Regards,
Mark.Zhan

art wrote:
> Hello Rongkai,
>
> Thanks! Good idea!
>
>
> ZR> After having a look at the latest 2.6.17-rc6 codes, __dev_alloc_skb is defined like this:
>
> ZR> #ifndef CONFIG_HAVE_ARCH_DEV_ALLOC_SKB
> ZR> /**
> ZR>  *      __dev_alloc_skb - allocate an skbuff for sending
> ZR>  *      @length: length to allocate
> ZR>  *      @gfp_mask: get_free_pages mask, passed to alloc_skb
> ZR>  *
> ZR>  *      Allocate a new &sk_buff and assign it a usage count of one. The
> ZR>  *      buffer has unspecified headroom built in. Users should allocate
> ZR>  *      the headroom they think they need without accounting for the
> ZR>  *      built in space. The built in space is used for optimisations.
> ZR>  *
> ZR>  *      %NULL is returned in there is no free memory.
> ZR>  */
> ZR> static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
> ZR>                                               gfp_t gfp_mask)
> ZR> {
> ZR>         struct sk_buff *skb = alloc_skb(length + NET_SKB_PAD, gfp_mask);
> ZR>         if (likely(skb))
> ZR>                 skb_reserve(skb, NET_SKB_PAD);
> ZR>         return skb;
> ZR> }
> ZR> #else
> ZR> extern struct sk_buff *__dev_alloc_skb(unsigned int length, int gfp_mask);
> ZR> #endif
>
>
> ZR> Therefore, you also can consider to implement your machine-specific __dev_alloc_skb() function, and force the skb is allocated from your low 32MB memory zone.
>
>
>
>