kernel addresses

kernel addresses

[Mark Lobo]

Hello!
I am an NT guy just getting into Linux stuff. Im going
to be writing a SCSI initiator driver for an FC HBA. 

Had a couple of simple questions.
1) What exactly is the difference between Kernel
logical and kernel virtual addresses? is the kernel
logical address the address space of the kernel? 
Seems like kernel logical address have a constant
offset from the physical address, and kernel virtual
addresses dont. Its slightly confusing on why have
these two separate terms at all? a kmalloced buffer
gives a logical address, but a vmalloced gives a
kernel virtual address. This kind of terminology
confused me!

2) If a SCSI command is passed by a user level app, is
there a buffer copy involved before it gets to the
scsi initiator? I saw something about bounce buffers,
but wasnt quite sure what it meant. So what are bounce
buffers used for? 
Does the original buffer allocated by the user get
mapped into a kernel virtual ( or logical???) address,
locked down and passed down the stack? 

Any help is greatly appreciated.

Thanks,
Mark



__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: kernel addresses

[Alan Cox]

On Mon, 2002-10-28 at 21:22, Mark Lobo wrote:
> Had a couple of simple questions.
> 1) What exactly is the difference between Kernel
> logical and kernel virtual addresses? is the kernel
> logical address the address space of the kernel? 
> Seems like kernel logical address have a constant
> offset from the physical address, and kernel virtual
> addresses dont. Its slightly confusing on why have
> these two separate terms at all? a kmalloced buffer
> gives a logical address, but a vmalloced gives a
> kernel virtual address. This kind of terminology
> confused me!

[Intel]
The kernel is mapped at 0xC0000000. It occupies a physically contiguous 
space somewhere aroud 0x100000. Memory you alloc with kmalloc comes out
of that mapping and is physically linear. So an 8K block consists of two
physically adjacent pages. Vmalloc allocates stuff by grabbing random
pages, mapping them again into the memory map and giving you something
which is seen as one linear range by virtual address but in fact
consists of multiple randomly distributed physical pages.

[General Purpose Answer]
You don't need to care. Anything you are using for DMA should go via the
PCI mapping API (see Documentation/DMA*). That will hand you back both
virtual addresses and DMA addresses.

> 2) If a SCSI command is passed by a user level app, is
> there a buffer copy involved before it gets to the
> scsi initiator? I saw something about bounce buffers,
> but wasnt quite sure what it meant. So what are bounce
> buffers used for? 
> Does the original buffer allocated by the user get
> mapped into a kernel virtual ( or logical???) address,
> locked down and passed down the stack? 

By the time it hits your SCSI driver it is locked into memory. Where it
came from you don't know and don't need to know. It may be locked user
memory it may be kernel space.

We use bounce buffers when something is at an address the card thinks it
cannot reach. Eg above 16Mb for ISA or above 4Gb for non DAC PCI cards

Re: kernel addresses

[Alan Cox]

kmalloc memory comes out of the pool requested  GFP_DMA selects the ISA
space, GFP_HIGH allows all 36bits (if I remember rightly). vmalloc comes
from the whole of memory space.

the highmem I/O stuff is a 2.4 patch some people use so that DMA aware
drivers that support the full 32bit DMA range can avoid bounce buffers. 

This comes from a 2.4 transition thing. Older drivers assume memory they
access is always mapped (eg when doing PIO) so the scsi code makes sure
this is true. The bounce patch lets drivers that don't do PIO avoid
this.

2.5 does the job properly instead as part of the block rewrite

Re: kernel addresses

[Mark Lobo]

Alan
Can vmalloced memory be used for DMA? an API will be
needed to get the physical address and sizes of the
segments of the buffer. Looks like virt_to_phys and
phys_to_virt work only on kernel "logical" addresses,
i.e. a constant offset difference. So is there any API
to get the physical addresses of the vmalloced
buffers?
( I havent dug into the DMA API in great detail, so
pardon me if that is exactly what it does)Also, Im
still not sure why we even need kernel logical
addresses separate from kernel "virtual" address? Why
not have just a single kernel virtual address?

Also, I still dont understand what is "low memory" and
"high memory". Why/how dont the older kernels support
high memory? 
 
Sorry if Im bothering you too much with these newbie
questions!

Thanks a lot!
Mark

--- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> kmalloc memory comes out of the pool requested 
> GFP_DMA selects the ISA
> space, GFP_HIGH allows all 36bits (if I remember
> rightly). vmalloc comes
> from the whole of memory space.
> 
> the highmem I/O stuff is a 2.4 patch some people use
> so that DMA aware
> drivers that support the full 32bit DMA range can
> avoid bounce buffers. 
> 
> This comes from a 2.4 transition thing. Older
> drivers assume memory they
> access is always mapped (eg when doing PIO) so the
> scsi code makes sure
> this is true. The bounce patch lets drivers that
> don't do PIO avoid
> this.
> 
> 2.5 does the job properly instead as part of the
> block rewrite
> 


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: kernel addresses

[Alan Cox]

On Mon, 2002-10-28 at 23:54, Mark Lobo wrote:
> Can vmalloced memory be used for DMA? an API will be

Simple answer is no. There are ways to do it but they are not nice
and shouldn't ever be needed. A couple of video capture drivers do
it because they need large internal buffers.

> Also, I still dont understand what is "low memory" and
> "high memory". Why/how dont the older kernels support
> high memory? 

Low/high is ambiguous so Im not actually sure. We have the following
limits

16Mb			DMA region 		(GFP_DMA)
900Mb (approximately)	Directly kernel mapped memory  (kmalloc/kernel
			page tables/pci mappings/vmalloc space etc)
4Gb			32bit PCI DMA limit
64Gb			Xeon memory limit

(We can't actually reach the 64Gb for other reasons)


Memory below the 900Mb boundary is directly accessible to the kernel,
memory above that is only mapped on demand. This is done because the
actual memory map looks like

--- 0 ----

Current user space application

---- 0xC0000000 ----

Kernel map to physical 0-> 900Mb

----- ?????? -----

vmalloc/ioremap space

---- 0xFFFFFFFF ----


Not having the user application memory mapped would require page table
changes whenever we make a system call - which is expensive. 

Re: kernel addresses

[Mark Lobo]

Alan,
So in that case, what exactly happens say when a user
app allocated a buffer for a write to a disk, and
passes it down to the top level driver. When he
mallocs that buffer, his address will be a user
address. And when it comes down to the initiator
driver it has to be a kernel address.  So when you say

" Not having the user application memory mapped would
require page table changes whenever we make a system
call - which is expensive.", does it mean all user
space addresses are permanently mapped to a kernel
address ( ioremapped )? If yes, how can that work,
because the mapping between the user virtual address
and the physical pages will be different at all times?
So how can we avoid building page tables again to get
a kernel view of the same memory?

Also, Im using a 2.4 kernel. So does this statement
still apply:
"When DMA I/O is performed to or from high memory, an
area is allocated in low memory known as a bounce
buffer. When data travels between a device and high
memory, it is first copied through the bounce buffer."
So if low memory is memory that can be addresses
directl y by the kernel, and high memory is say a user
allocated buffer (?), do I still have to do what the
patch wants me to do? When does he mid layer decide in
this case to allocate a bounce buffer? In this case is
the "low" and "high" limit the ISA limit that the
driver indicates in the host template structure while
initialization?

Thanks!
Mark

--- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> On Mon, 2002-10-28 at 23:54, Mark Lobo wrote:
> > Can vmalloced memory be used for DMA? an API will
> be
> 
> Simple answer is no. There are ways to do it but
> they are not nice
> and shouldn't ever be needed. A couple of video
> capture drivers do
> it because they need large internal buffers.
> 
> > Also, I still dont understand what is "low memory"
> and
> > "high memory". Why/how dont the older kernels
> support
> > high memory? 
> 
> Low/high is ambiguous so Im not actually sure. We
> have the following
> limits
> 
> 16Mb			DMA region 		(GFP_DMA)
> 900Mb (approximately)	Directly kernel mapped memory 
> (kmalloc/kernel
> 			page tables/pci mappings/vmalloc space etc)
> 4Gb			32bit PCI DMA limit
> 64Gb			Xeon memory limit
> 
> (We can't actually reach the 64Gb for other reasons)
> 
> 
> Memory below the 900Mb boundary is directly
> accessible to the kernel,
> memory above that is only mapped on demand. This is
> done because the
> actual memory map looks like
> 
> --- 0 ----
> 
> Current user space application
> 
> ---- 0xC0000000 ----
> 
> Kernel map to physical 0-> 900Mb
> 
> ----- ?????? -----
> 
> vmalloc/ioremap space
> 
> ---- 0xFFFFFFFF ----
> 
> 
> Not having the user application memory mapped would
> require page table
> changes whenever we make a system call - which is
> expensive. 
> 
> 
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
http://vger.kernel.org/majordomo-info.html


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

Re: kernel addresses

[Alan Cox]

On Tue, 2002-10-29 at 00:36, Mark Lobo wrote:
> So in that case, what exactly happens say when a user
> app allocated a buffer for a write to a disk, and
> passes it down to the top level driver. When he
> mallocs that buffer, his address will be a user
> address. And when it comes down to the initiator
> driver it has to be a kernel address.  So when you say

It might be a mapped user buffer, it might be kernel allocated, you
don't know, you can't tell, you don't need to tell. You get given a
valid kernel address.

> Also, Im using a 2.4 kernel. So does this statement
> still apply:
> "When DMA I/O is performed to or from high memory, an
> area is allocated in low memory known as a bounce
> buffer. When data travels between a device and high
> memory, it is first copied through the bounce buffer."
> So if low memory is memory that can be addresses
> directl y by the kernel, and high memory is say a user
> allocated buffer (?), do I still have to do what the
> patch wants me to do? When does he mid layer decide in
> this case to allocate a bounce buffer? In this case is
> the "low" and "high" limit the ISA limit that the
> driver indicates in the host template structure while
> initialization?

The ISA limit if the template says unchecked_isa_dma: 1, otherwise the
low memory limit. On 2.5 its done by the pci limits and instead of an
address you get passed a 32bit page number and offset (so you can do
64bit DMA sanely)

Re: kernel addresses

[Alan Cox]

For the 2.4 case you will always get kernel addresses. Bounce buffer
handling will depend where the page being referenced lives and if its
directly kernel accessible. In the 2.5 case it depends on your pci
device mask