Does kmalloc always return address below 4GB?

Does kmalloc always return address below 4GB?

[Adam J. Richter]

	I am trying to convert linux-2.5.6-pre2/drivers/scsi/advansys.c
to the DMA mapping system described in Documenation/DMA-mapping.txt
in an effort to try to help to make all of the drivers compile again.
I have a question that would help me with this process, and probably
others who has occasion to fiddle with any of the many devices covered
by this question.

	According to DMA-mapping.txt, I am allowed to use use
an address returned from kmalloc as a DMA address, without
need to convert by pci_map_single or allocate space with
pci_alloc_consistent.  The advansys.c hardware (and probably
many PCI devices) asssumes 32-bit addresses.  Can I rely on
on all architectures that support PCI or ISA that kmalloc
will always return an address below 2**32 if GFP_HIGHMEM
is not specified?  If not, then I guess I can use
pci_alloc_consistent and pci_map_single as necessary, since they
should can potentially know that I am using a device that
only understands 32-bit addresses, from my earlier call to
pci_set_dma_mask.  However, I assume that it is considered
simpler and therefore better to avoid these routines when possible.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: "Adam J. Richter" <adam@yggdrasil.com>
   Date: Tue, 5 Mar 2002 03:12:52 -0800

   If not, then I guess I can use
   pci_alloc_consistent and pci_map_single as necessary, since they
   should can potentially know that I am using a device that
   only understands 32-bit addresses, from my earlier call to
   pci_set_dma_mask.  However, I assume that it is considered
   simpler and therefore better to avoid these routines when possible.

Just use pci_alloc_consistent, it never gives you
anything larger than 32-bit addresses, please read the
documentation :-)

On 64-bit platforms without CONFIG_HIGHMEM, kmalloc can return any
pointer, but that is fine since your DMA mask will instruct the
IOMMU layer of the platform to map it in the low 32-bits of DMA
address space.

If you follow DMA-mapping.txt to the letter, it will just work, trust
us. :-)

Re: Does kmalloc always return address below 4GB?

[Steffen Persvold]

"David S. Miller" wrote:
> 
>    From: "Adam J. Richter" <adam@yggdrasil.com>
>    Date: Tue, 5 Mar 2002 03:12:52 -0800
> 
>    If not, then I guess I can use
>    pci_alloc_consistent and pci_map_single as necessary, since they
>    should can potentially know that I am using a device that
>    only understands 32-bit addresses, from my earlier call to
>    pci_set_dma_mask.  However, I assume that it is considered
>    simpler and therefore better to avoid these routines when possible.
> 
> Just use pci_alloc_consistent, it never gives you
> anything larger than 32-bit addresses, please read the
> documentation :-)
> 

What about memory for streaming mappings. I know pci_map_single (and _sg) will
use bounce buffers on platforms without an IOMMU, but wouldn't it be nice if
this could also be ensured by the device driver to avoid uneccessary copying.
It could for example be solved with a GFP_32BIT flag or something (on IA64 I
know GFP_DMA is used in pci_alloc_consistent() to get memory below 4GB but that
can't be used on all platforms).


> On 64-bit platforms without CONFIG_HIGHMEM, kmalloc can return any
> pointer, but that is fine since your DMA mask will instruct the
> IOMMU layer of the platform to map it in the low 32-bits of DMA
> address space.
> 
> If you follow DMA-mapping.txt to the letter, it will just work, trust
> us. :-)

Regards,
-- 
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best
 mailto:sp@scali.com |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.13.8 -
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >320MBytes/s and <4uS latency

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: Steffen Persvold <sp@scali.com>
   Date: Tue, 05 Mar 2002 16:43:01 +0100

   "David S. Miller" wrote:
   > Just use pci_alloc_consistent, it never gives you
   > anything larger than 32-bit addresses, please read the
   > documentation :-)
   
   What about memory for streaming mappings

That's a different story.

   I know pci_map_single (and _sg) will
   use bounce buffers on platforms without an IOMMU,

64-bit platforms without IOMMU use HIGHMEM.

   It could for example be solved with a GFP_32BIT flag or something (on IA64 I
   know GFP_DMA is used in pci_alloc_consistent() to get memory below 4GB but that
   can't be used on all platforms).
   
IA64 was broken, it now uses HIGHMEM.
   
Franks a lot,
David S. Miller
davem@redhat.com

Re: Does kmalloc always return address below 4GB?

[David Mosberger]

>>>>> On Tue, 05 Mar 2002 07:47:22 -0800 (PST), "David S. Miller" <davem@redhat.com> said:

  DaveM>    I know pci_map_single (and _sg) will use bounce buffers on
  DaveM> platforms without an IOMMU,

  DaveM> 64-bit platforms without IOMMU use HIGHMEM.

Not true for ia64 linux.

  DaveM>    It could for example be solved with a GFP_32BIT flag or
  DaveM> something (on IA64 I know GFP_DMA is used in
  DaveM> pci_alloc_consistent() to get memory below 4GB but that can't
  DaveM> be used on all platforms).

  DaveM> IA64 was broken, it now uses HIGHMEM.

No it doesn't.  Perhaps you meant to say that the Red Hat version of
the ia64 linux kernel uses highmem?

	--david

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: David Mosberger <davidm@napali.hpl.hp.com>
   Date: Tue, 5 Mar 2002 08:44:18 -0800

   >>>>> On Tue, 05 Mar 2002 07:47:22 -0800 (PST), "David S. Miller" <davem@redhat.com> said:
     DaveM> IA64 was broken, it now uses HIGHMEM.
   
   No it doesn't.  Perhaps you meant to say that the Red Hat version of
   the ia64 linux kernel uses highmem?

Oh I didn't know you didn't use those changes.
Your version is still broken then.

Re: Does kmalloc always return address below 4GB?

[David Mosberger]

>>>>> On Tue, 05 Mar 2002 17:09:09 -0800 (PST), "David S. Miller" <davem@redhat.com> said:

  DaveM> Oh I didn't know you didn't use those changes.

No we don't.  We discussed it but the feeling was that the swiotlb is
overall the better choice (for 2.4.xx).  I don't really care about
this issue all that much myself, as it affects only platforms that
don't have a hardware I/O TLB and use PCI cards that can't address the
entire physical address space.

  DaveM> Your version is still broken then.

Ah, classic DaveM.  You're obviously entitled to your opinion.

	--david

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: David Mosberger <davidm@napali.hpl.hp.com>
   Date: Tue, 5 Mar 2002 17:31:33 -0800
   
     DaveM> Your version is still broken then.
   
   Ah, classic DaveM.  You're obviously entitled to your opinion.

Look at all the problems that swiotlb pops up.  I'm not talking about
"can we make it work", but "can we make the performance not suck bad
when the thing is actually used".

IMHO highmem is the only clean and good performing solution for
>4GB machines without IOMMU.

You show me how to make swiotlb faster and more pretty, than we
can talk. :-)

So when someone tells me "on ia64, with 8gb ram, my eepro100 card gets
really crap performance under any load" I will explain the above to
them and let them know that the performance sucks because the ia64
folks refuse to integrate this bug fix :-)

Re: Does kmalloc always return address below 4GB?

[David Mosberger]

>>>>> On Tue, 05 Mar 2002 17:52:38 -0800 (PST), "David S. Miller" <davem@redhat.com> said:

  DaveM> So when someone tells me "on ia64, with 8gb ram, my eepro100
  DaveM> card gets really crap performance under any load" I will
  DaveM> explain the above to them and let them know that the
  DaveM> performance sucks because the ia64 folks refuse to integrate
  DaveM> this bug fix :-)

I think you're ignoring disk I/O.

I'm not terribly interested in revisiting this topic.  If you care,
take it up with the folks that build the chips without hardware I/O
TLB.  They are the ones with the vested interest. ;-)

	--david

Re: Does kmalloc always return address below 4GB?

[Adam J. Richter]

>Just use pci_alloc_consistent, it never gives you
>anything larger than 32-bit addresses, please read the
>documentation :-)

	I see the smiley, but let me point out that I have
read Documentation/DMA-mapping.txt and I was misled by this
sentence:

| If you acquired your memory via the page allocator
| (i.e. __get_free_page*()) or the generic memory allocators
| (i.e. kmalloc() or kmem_cache_alloc()) then you may DMA to/from
| that memory using the addresses returned from those routines.

	It might be a good idea to rephrase it.  If I knew what that
sentence I would propose a patch to the DMA-mapping.txt file, but I
honestly don't know what proposition that sentence is supposed
to convey.  If there really is no guarantee that this sentence is
conveying, then I guess the sentence should be deleted.

	Anyhow, thanks for your quick clarification.  The driver
breaking on 64-bit architectures was exactly what I was worried about.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: "Adam J. Richter" <adam@yggdrasil.com>
   Date: Tue, 5 Mar 2002 03:52:14 -0800
   
   | If you acquired your memory via the page allocator
   | (i.e. __get_free_page*()) or the generic memory allocators
   | (i.e. kmalloc() or kmem_cache_alloc()) then you may DMA to/from
   | that memory using the addresses returned from those routines.
   
   	It might be a good idea to rephrase it.  If I knew what that
   sentence I would propose a patch to the DMA-mapping.txt file, but I
   honestly don't know what proposition that sentence is supposed
   to convey.  If there really is no guarantee that this sentence is
   conveying, then I guess the sentence should be deleted.

Probably it should qualify what it means with "and you used
GFP_KERNEL".  Because that was the intention.

I'll fix that.

However, you can use GFP_HIGH memory with pci_map_page _iff_
you set your DMA mask to allow 64-bits.

The original impetus for that quoted bit of DMA-mapping.txt
was to make sure nobody used vmalloc() or kmap() pointers.

Re: Does kmalloc always return address below 4GB?

[Adam J. Richter]

>   | If you acquired your memory via the page allocator
>   | (i.e. __get_free_page*()) or the generic memory allocators
>   | (i.e. kmalloc() or kmem_cache_alloc()) then you may DMA to/from
>   | that memory using the addresses returned from those routines.
[...]
>Probably it should qualify what it means with "and you used
>GFP_KERNEL".  Because that was the intention.

	That change alone would result in a setence that
I still would not understand.  Does GFP_KERNEL guarantee address
below 4GB on all architectures?  The GFP_KERNEL mask that currently
includes __GFP_IO, but I'm not sure that that even means "below 4GB"
on all architectures.  If __GFP_IO really does guarantee "below 4GB
and physical = virtual" on all architectures, then it sounds like
I am back to not having to use pci_{,un}map_single in a number of
places in advansys.c, because the Scsi_Request structurs are allocated
with GFP_ATOMIC, which also includes __GFP_IO.

>I'll fix that.

>However, you can use GFP_HIGH memory with pci_map_page _iff_
>you set your DMA mask to allow 64-bits.

>The original impetus for that quoted bit of DMA-mapping.txt
>was to make sure nobody used vmalloc() or kmap() pointers.

	Just to be clear, I assume that you mean that you cannot
simply cast these virtual addresses to dma_addr_t and that the
underlying physical memory is not guaranteed to be below 4GB,
but that you can use that memory with pci_map_single if your
PCI device can handle 64 bit addresses.

	If I got it right, then here is some proposed replacement
text, to possibly save you a little effort:

| 	You cannot directly use an address in an area returned by
| vmalloc() or kmap(), because these routines are *not* guaranteed to
| return addresses where virtual == cpu memory bus == pci.  You
| cannot even use those addresses with PCI device that only have
| 32 bit addressing, because the underlying physical memory may be
| above 4GB.
|
| 	However, a PCI card that does 64-bit addressing can
| access memory returned by vmalloc or kmap via the dma_addr_t
| address returned by pci_map_single or pci_map_page.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: "Adam J. Richter" <adam@yggdrasil.com>
   Date: Tue, 5 Mar 2002 06:43:35 -0800
   
   	Just to be clear, I assume that you mean that you cannot
   simply cast these virtual addresses to dma_addr_t and that the
   underlying physical memory is not guaranteed to be below 4GB,
   but that you can use that memory with pci_map_single if your
   PCI device can handle 64 bit addresses.
   
   	If I got it right, then here is some proposed replacement
   text, to possibly save you a little effort:

You haven't got it right.  Physical address > 4GB does not mean
your 32-bit device cannot DMA to it.  Stop thinking about
implementation, that's the whole point of the abstraction :-)

On 64-bit platforms that don't set CONFIG_HIGHMEM, they have MMU's on
the PCI bus that can map arbitrary 64-bit physical addresses to 32-bit
PCI bus addresses.  So on these platforms you may pass any pointer
from kmalloc()/alloc_page() whatsoever into the pci_map_foo()
routines.

In order to handle highmem pages, you have to set your DMA mask
appropriately (to indicate 64-bit addressing capability) and
use pci_map_page() instead of pci_map_single().

Look at other drivers using the DMA interfaces like the two aic7xxx
and all of the sym53c8xx drivers, they get it right.

Re: Does kmalloc always return address below 4GB?

[Adam J. Richter]

>   From: "Adam J. Richter" <adam@yggdrasil.com>
>   Date: Tue, 5 Mar 2002 06:43:35 -0800
>   
>   	Just to be clear, I assume that you mean that you cannot
>   simply cast these virtual addresses to dma_addr_t and that the
>   underlying physical memory is not guaranteed to be below 4GB,
>   but that you can use that memory with pci_map_single if your
>   PCI device can handle 64 bit addresses.
>   
>   	If I got it right, then here is some proposed replacement
>   text, to possibly save you a little effort:

>You haven't got it right.  Physical address > 4GB does not mean
>your 32-bit device cannot DMA to it.  Stop thinking about
>implementation, that's the whole point of the abstraction :-)

>On 64-bit platforms that don't set CONFIG_HIGHMEM, they have MMU's on
>the PCI bus that can map arbitrary 64-bit physical addresses to 32-bit
>PCI bus addresses.  So on these platforms you may pass any pointer
>from kmalloc()/alloc_page() whatsoever into the pci_map_foo()
>routines.

	I think you're confusing a "there exists one x" with "for every
x".  In your statement, you've reduced your universe to "platforms
that don't set CONFIG_HIGHMEM", but that is not all systems.

	While there exists one (or more) computers that have this
reverse mapping hardware, I believe that it is not true of *all*
platforms with >4GB of RAM.  I infer from your statement that
all 64-bit platforms that lack this hardware should define CONFIG_HIGHMEM.

>In order to handle highmem pages, you have to set your DMA mask
>appropriately (to indicate 64-bit addressing capability) and
>use pci_map_page() instead of pci_map_single().

	Let's say I have some random Pentium3 or Pentium4
machine with >4GB of memory, the PCI card in question only does 32 bit
addressing.  This is a CONFIG_HIGMEM platform and it lacks the
MMU that you discuss in the first paragraph of DMA-mapping.txt, right?

	Now imagine that a vmalloc in this driver returns a page
above 4GB (include/linux/vmalloc.h defines vmalloc() to pass __GFP_HIGHMEM).
What will pci_map_single return?   I think, under x86, pci_map_single
will call __pa, which will return the underlying physical address,
which, in this case, would be above 4GB, which would not be accessible
by the PCI card.

>Look at other drivers using the DMA interfaces like the two aic7xxx
>and all of the sym53c8xx drivers, they get it right.

	Grepping for vmalloc and kmap in them turns up no hits.

	I understand that your pci_alloc_consistent abstration allows
one to write a driver for a 32-bit PCI card that, on 64-bit systems
with the MMU that you describe, that will be able to use memory above 4GB
for IO transfers, like so:

		pci_set_dma_mask(pcidev, 0xffffffff);
		addr = pci_alloc_consistent(pcidev, nbytes, direction,
					    &dma_addr);
		/* __pa(addr) may be >4GB, but only on systems with
		   PCI address mapping hardware.  dma_address will
		   be <4GB on all systems. */
		TELL_DEVICE_TO_DO_TRANSFER(dma_addr, nbytes);
		pci_free_consistent(...);

	Maybe I need to rephrase my proposed text for greater
clarity.  The point of my proposed text was that, in the absense of
"#ifndef CONFIG_HIGMEM", the following code will not work on a 32-bit
computer with >4GB of RAM (CONFIG_HIGHEM) talking to a PCI card
that only does 32-bit addressing:

		pci_set_dma_mask(pcidev, 0xffffffff);
		addr = vmalloc(nbytes);
		/* On an x86 with >4GB of RAM, addr will be <4GB, but
	           __pa(addr) might be >4GB, and the system lacks
	           PCI address mapping harware. */

		dma_addr = pci_map_single(pcidev, addr, nbytes, direction);
		/* Uh oh!  dma_addr may be >4GB and I might not have	
		   PCI address mapping hardware! */
		TELL_DEVICE_TO_DO_TRANSFER(dma_addr, nbytes);
		pci_unmap_single(...);

	Was this unclear in my proposed text or do I still misunderstand
some fact that you're trying to convey (if so, sorry if for apparently
being so dense about it)?

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

Re: Does kmalloc always return address below 4GB?

[Gerd Knorr]

> >Look at other drivers using the DMA interfaces like the two aic7xxx
> >and all of the sym53c8xx drivers, they get it right.
>  
>  	Grepping for vmalloc and kmap in them turns up no hits.

Why do you want to vmalloc() memory in the scsi driver?

>  computer with >4GB of RAM (CONFIG_HIGHEM) talking to a PCI card
>  that only does 32-bit addressing:
>  
>  		pci_set_dma_mask(pcidev, 0xffffffff);
>  		addr = vmalloc(nbytes);
>  		/* On an x86 with >4GB of RAM, addr will be <4GB, but
>  	           __pa(addr) might be >4GB, and the system lacks
>  	           PCI address mapping harware. */

use vmalloc_32(), this one returns lowmem.

>  		dma_addr = pci_map_single(pcidev, addr, nbytes, direction);

This is illegal because addr is a kernel _virtual_ address.  You have to
get the page using vmalloc_to_page() and feed this to pci_map_page()
then.  With nbytes > PAGE_SIZE you probably want to build a scatterlist
and use pci_map_sg().

  Gerd

-- 
#include </dev/tty>

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: Gerd Knorr <kraxel@bytesex.org>
   Date: 5 Mar 2002 17:00:55 GMT
   
   >  computer with >4GB of RAM (CONFIG_HIGHEM) talking to a PCI card
   >  that only does 32-bit addressing:
   >  
   >  		pci_set_dma_mask(pcidev, 0xffffffff);
   >  		addr = vmalloc(nbytes);
   >  		/* On an x86 with >4GB of RAM, addr will be <4GB, but
   >  	           __pa(addr) might be >4GB, and the system lacks
   >  	           PCI address mapping harware. */
   
   use vmalloc_32(), this one returns lowmem.

You can't use vmalloc() pointers as arguments to pci_map_single.
That is the point of what he's mentioning.
   
   >  		dma_addr = pci_map_single(pcidev, addr, nbytes, direction);
   
   This is illegal because addr is a kernel _virtual_ address.  You have to
   get the page using vmalloc_to_page() and feed this to pci_map_page()
   then. 

There is no such requirement that pci_map_page() only be used.
If you know you haven't got a HIGHMEM page, it is legal to
pass this to pci_map_single().

Re: Does kmalloc always return address below 4GB?

[Steffen Persvold]

"Adam J. Richter" wrote:
> 
[snip]
> 
>         I understand that your pci_alloc_consistent abstration allows
> one to write a driver for a 32-bit PCI card that, on 64-bit systems
> with the MMU that you describe, that will be able to use memory above 4GB
> for IO transfers, like so:
> 
>                 pci_set_dma_mask(pcidev, 0xffffffff);
>                 addr = pci_alloc_consistent(pcidev, nbytes, direction,
>                                             &dma_addr);
>                 /* __pa(addr) may be >4GB, but only on systems with
>                    PCI address mapping hardware.  dma_address will
>                    be <4GB on all systems. */
>                 TELL_DEVICE_TO_DO_TRANSFER(dma_addr, nbytes);
>                 pci_free_consistent(...);
> 
>         Maybe I need to rephrase my proposed text for greater
> clarity.  The point of my proposed text was that, in the absense of
> "#ifndef CONFIG_HIGMEM", the following code will not work on a 32-bit
> computer with >4GB of RAM (CONFIG_HIGHEM) talking to a PCI card
> that only does 32-bit addressing:
> 
>                 pci_set_dma_mask(pcidev, 0xffffffff);
>                 addr = vmalloc(nbytes);
>                 /* On an x86 with >4GB of RAM, addr will be <4GB, but
>                    __pa(addr) might be >4GB, and the system lacks
>                    PCI address mapping harware. */
> 
>                 dma_addr = pci_map_single(pcidev, addr, nbytes, direction);
>                 /* Uh oh!  dma_addr may be >4GB and I might not have
>                    PCI address mapping hardware! */
>                 TELL_DEVICE_TO_DO_TRANSFER(dma_addr, nbytes);
>                 pci_unmap_single(...);
> 
>         Was this unclear in my proposed text or do I still misunderstand
> some fact that you're trying to convey (if so, sorry if for apparently
> being so dense about it)?
> 


This is sort of the same question I have, the only problem you will have here is that vmalloc() will
return only _virtual_ contiguous pages, not physical, so you would actually have to use pci_map_sg()
instead of pci_map_single(). The problem is that vmalloc() itself is not restricted to allocate
pages which is guaranteed to be directly DMA capable for the PCI device so the pci_map_xxx function
will have to allocate bounce buffers for the data if the physical address is not within the device's
limits. I'm proposing a new interface, something in the line of :

First set the DMA boundaries for our device:
pci_set_dma_mask(pcidev, 0xffffffff);

Then use this API to allocate DMA capable memory (this API also does the mapping to PCI so the
pci_map_xx calls is not needed when using it) :

dmamem = pci_alloc_dmamem(pcidev, nbytes, &vaddr, CONSISTENT);

to get consistent memory (a memory region where caching and so on would be turned off for certain
platforms). This memory is of course physical contiguous (this is the equivalent to the existing
pci_alloc_consistent() function).

dmamem = pci_dmamem_alloc(pcidev, nbytes, &vaddr, BIDIRECTIONAL);

to get a streaming memory region which should be accessible from kernel space, but isn't needed to
be physical contiguous (i.e. using a scatter-gather table for all the physical pages when mapping it
to PCI). vmalloc() could be used to get the pages here.

dmamem = pci_dmamem_alloc(pcidev, nbytes, &vaddr, BIDIRECTIONAL | CONTIGUOUS);

to get a streaming memory region which should be accessible from kernel space and also physical
contiguous (i.e. using get_free_pages() or kmalloc() to get the pages).

dmamem = pci_dmamem_alloc(pcidev, nbytes, NULL, BIDIRECTIONAL);

to get a streaming memory region which is not accessed by the kernel at all (i.e a frame grabber
buffer or a SCI shared memory segment only used in user space).

Feeding the I/O addresss and length to the actual PCI adapter should be done sort of the same way as
before :

nents = pci_dmamem_nents(dmamem);
for (i = 0; i < nents; i++) {
   hw_address[i] = pci_dmamem_address(dmamem, i);
   hw_len[i] = pci_dmamem_len(dmamem, i);
}

On contigous and consistent memory regions, nents should be one and therefore no looping should be
neccessary :

hw_address = pci_dmamem_address(dmamem, 0);
hw_len = pci_dmamem_len(dmamem, 0);

hw_len here should of course correspond to the nbytes argument given to the pci_alloc_dmamem()
function.

So, what do you think ? Is this something we should think of for 2.5, or am I on the wrong side of
the road here ?

Regards,
-- 
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best
 mailto:sp@scali.com |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.13.8 -
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >320MBytes/s and <4uS latency

Re: Does kmalloc always return address below 4GB?

[Adam J. Richter]

Steffen Persvold writes:
>I know pci_map_single (and _sg) will
>use bounce buffers on platforms without an IOMMU [...]

	For a moment I thought that must be the point that I
was missing, but I don't see any such bounce buffer support
in linux-2.5.6-pre2/include/asm-i386/pci.h or arch/i386/kernel/pci-dma.c.
I do not see how this is currently implemented on x86 systems with >4GB
of RAM.

	Or, by "will use" did you mean "will implement in the future?"

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

Re: Does kmalloc always return address below 4GB?

[David S. Miller]

   From: "Adam J. Richter" <adam@yggdrasil.com>
   Date: Tue, 5 Mar 2002 09:08:20 -0800

   Steffen Persvold writes:
   >I know pci_map_single (and _sg) will
   >use bounce buffers on platforms without an IOMMU [...]
   
   	For a moment I thought that must be the point that I
   was missing, but I don't see any such bounce buffer support
   in linux-2.5.6-pre2/include/asm-i386/pci.h or arch/i386/kernel/pci-dma.c.
   I do not see how this is currently implemented on x86 systems with >4GB
   of RAM.

You won't get HIGHMEM pages in your driver unless you are using
2.5.x and tell the scsi layer you are capable to DMA to/from
HIGHMEM pages.

Similarly for networking drivers, and setting NETIF_F_HIGHDMA in
the net device feature flags.

Re: Does kmalloc always return address below 4GB?

[Adam J. Richter]

>From: Steffen Persvold <sp@scali.com>
[...]
>This is sort of the same question I have, the only problem you will
>have here is that vmalloc() will
>return only _virtual_ contiguous pages [...]

	In my example, you can assume the amount of memory
being allocated fits into a single page.  I was not talking
about issues related to crossing page boundaries.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."