Kernel memory

Kernel memory

[Gilles Pokam]

Hi,

I'm not sure this is the right place to post my message, but I do need
some advice for my problem.

I have a special user application who needs to access any part of the
kernel memory. My question is therefore how to make the whole memory
accessible for that particular application ?

Thanks.

Re: Kernel memory

[Chris Wedgwood]

On Thu, Apr 28, 2005 at 04:52:21PM -0700, Gilles Pokam wrote:

> I have a special user application who needs to access any part of
> the kernel memory.

why & for what?

> My question is therefore how to make the whole memory accessible for
> that particular application ?

maybe /dev/kmem or /proc/kcore

it would help if you explain in more detail what you are trying to do

Re: Kernel memory

[Gilles Pokam]

On 4/28/05, Chris Wedgwood <cw@f00f.org> wrote:
> On Thu, Apr 28, 2005 at 04:52:21PM -0700, Gilles Pokam wrote:
> 
> > I have a special user application who needs to access any part of
> > the kernel memory.
> 
> why & for what?
> 
> > My question is therefore how to make the whole memory accessible for
> > that particular application ?
> 
> maybe /dev/kmem or /proc/kcore
> 
> it would help if you explain in more detail what you are trying to do
> 

Here is the big picture of the what (detailed are omitted):
I'm experimenting with system to help developers debug their programs.
The debugger is based on a replayer whose purpose is to
deterministically replay the last millions of instructions that lead
to a crash. On the user site, when the program runs, a trace of load
values along with some architectural state is continuously recorded.
Upon a crash, this trace is sent to the developer for debugging. At
the developer site, the application is replayed as follows. The
architectural state is initialized by reading the values from the
trace. The execution then proceeds as follows. If a load instruction
is encountered, the value is taken from the trace. Otherwise all other
instructions execute normally.

Issues:
Now, there are cases where a load value read from the trace (a virtual
address) raises a pagefault exception at the execution because the
address is invalid (the OS has not mapped this address) or the
accessed page doesn't have the right protection. In such cases,
instead of having the program to segfault, I want to handle this as a
valid pagefault, returning a page at the place. One other constraint
is that the application that is being debugged must not be modified.

I was thinking of making the whole memory accessible to handle this.
But I can not rely on mapping /dev/mem or /proc/kcore into the user
space since this would require modifying the binary. Are there other
ways of doing this ? May be disabling paging ? if so, how to do this ?


Thanks for any suggestions.

Gilles

Re: Kernel memory

[Chris Wedgwood]

On Thu, Apr 28, 2005 at 10:33:16PM -0700, Gilles Pokam wrote:

> I was thinking of making the whole memory accessible to handle this.
> But I can not rely on mapping /dev/mem or /proc/kcore into the user
> space since this would require modifying the binary. Are there other
> ways of doing this ? May be disabling paging ? if so, how to do this
> ?

why can't you use a wrapper?

Re: Kernel memory

[Gilles Pokam]

On 4/28/05, Chris Wedgwood <cw@f00f.org> wrote:
> On Thu, Apr 28, 2005 at 10:33:16PM -0700, Gilles Pokam wrote:
> 
> > I was thinking of making the whole memory accessible to handle this.
> > But I can not rely on mapping /dev/mem or /proc/kcore into the user
> > space since this would require modifying the binary. Are there other
> > ways of doing this ? May be disabling paging ? if so, how to do this
> > ?
> 
> why can't you use a wrapper?

Can you be more explicit ?

Thanks.

Gilles

Re: Kernel memory

[Chris Wedgwood]

On Thu, Apr 28, 2005 at 10:48:31PM -0700, Gilles Pokam wrote:

> Can you be more explicit ?

why can't you have the parent process of whatever your tracing mess
with /dev/kmem or whatever so you don't have to frob the original
binary?

i guess it's not really clear to me what you're doing entirely

Re: Kernel memory

[Gilles Pokam]

On 4/28/05, Chris Wedgwood <cw@f00f.org> wrote:
> On Thu, Apr 28, 2005 at 10:48:31PM -0700, Gilles Pokam wrote:
> 
> > Can you be more explicit ?
> 
> why can't you have the parent process of whatever your tracing mess
> with /dev/kmem or whatever so you don't have to frob the original
> binary?

I see the point. Just to test this solution, I tried before to modify
a test application by mmaping /dev/mem into the application address
range. Since I don't know apriori which address is going to raise a
pagefault, I had to mmap the entire memory to the user space. However
this doesn't work. It looks like there is a limitation on the amount
of memory you can mmap ?

> i guess it's not really clear to me what you're doing entirely

the simplest way to say is: I want the pagefault handler to return a
memory page when it encounters a pagefault exceptions due to an
invalid address or incorrect page protection.


Thanks. 
Gilles

Re: Kernel memory

[Chris Wedgwood]

On Thu, Apr 28, 2005 at 11:45:40PM -0700, Gilles Pokam wrote:

> the simplest way to say is: I want the pagefault handler to return a
> memory page when it encounters a pagefault exceptions due to an
> invalid address or incorrect page protection.

where should this page come from?

Re: Kernel memory

[Gilles Pokam]

On 4/28/05, Chris Wedgwood <cw@f00f.org> wrote:
> On Thu, Apr 28, 2005 at 11:45:40PM -0700, Gilles Pokam wrote:
> 
> > the simplest way to say is: I want the pagefault handler to return a
> > memory page when it encounters a pagefault exceptions due to an
> > invalid address or incorrect page protection.
> 
> where should this page come from?

One way is to make the pagefault handler return a new vma that
contains the faulty address when such a scenario is encountered. The
normal pagefault mechanism will then apply by the time that address
gets accessed again since the address will now be valid. So a page
will be allocated. But I don't know how what should be changed  in the
kernel to enable this behavior.

Kernel Memory

[Vijay Chauhan]

Hello,

I am newbie.
It has been said "kernel memory is not pageable"
What does it mean? There is no concept of kernel virtual address?

Any simple explanation will help me to udnerstand.

Thanks,
Vijay

Kernel Memory

[HowKernel StuffWorks]

Hi,
  I am a kernel newbie too, Kernel memory is not pagable, because kernel
itself is responsible for paging. See this discussion,
http://kerneltrap.org/node/6404

  Paging happens for regular processes, i.e each process memory is divided
into a page of certain size(4kb in Linux), so it can swap for another page
that might be needed for that particular moment, it usually replaces an
page that have not been used for long time(see LRU,internal fragmentation).
Kernel is a process too, just like anyother process, but it differs from
others because it directly talks to the hardware and also it is the one
which takes care of paging and LRU algo's. So it does not make much sense,
for a kernel memory to be pageable, because, if it removes the page that
contains that has the paging algorithm, then there is  no way to come back,
you want to retrieve a page from disk_swap_area, but you cannot, because
the page that contains code to retrieve the instructions are paged :)...
This is not just for paging, since kernel controls everything, it is not
advisable to put its own code in swap area(see, virtual memory, virtual
memory = physical memory + swap space).

BTW, i am a kernel newbie too, this is my basic understanding, please feel
free to correct it, if I am wrong....

On Thu, Jun 21, 2012 at 8:27 AM, Vijay Chauhan <kernel.vijay@gmail.com>wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120621/282abbb9/attachment-0001.html 

Kernel Memory

[kishore sheik ahamed]

Hey Vijay

I am a newbie too. Just sharing what I could go through.

It is said that Kernel or atleast a part of kernel needs to be non paged
for fast interrupt access etc as pinned memory
Wiki says
Pinned/Locked/Fixed pages

Operating systems have memory areas that are pinned (never swapped to
secondary storage). For example,
interrupt<http://en.wikipedia.org/wiki/Interrupt>mechanisms rely on an
array of pointers to their handlers, such as
I/O <http://en.wikipedia.org/wiki/I/O> completion and page
fault<http://en.wikipedia.org/wiki/Page_fault>.
If the pages containing these pointers or the code that they invoke were
pageable, interrupt-handling would become far more complex and
time-consuming, particularly in the case of page fault interrupts. Hence,
some part of the page table structures is not pageable.

Some pages may be pinned for short periods of time, others may be pinned
for long periods of time, and still others may need to be permanently
pinned. For example:

   - The paging supervisor code and drivers for secondary storage devices
   on which pages reside must be permanently pinned, as otherwise paging
   wouldn't even work because the necessary code wouldn't be available.
   - Timing-dependent components may be pinned to avoid variable paging
   delays.
   - Data buffers <http://en.wikipedia.org/wiki/Data_buffer> that are
   accessed directly by peripheral devices that use direct memory
access<http://en.wikipedia.org/wiki/Direct_memory_access>or I/O
   channels <http://en.wikipedia.org/wiki/I/O_channel> must reside in
   pinned pages while the I/O operation is in progress because such devices
   and the buses <http://en.wikipedia.org/wiki/Bus_%28computing%29> to
   which they are attached expect to find data buffers located at physical
   memory addresses; regardless of whether the bus has a memory management
   unit for I/O <http://en.wikipedia.org/wiki/IOMMU>, transfers cannot be
   stopped if a page fault occurs and then restarted when the page fault has
   been processed.

There are other two discussion thread which say kernel is non-pageable and
now due to growing kernel Data structures it is allowed

http://kerneltrap.org/node/6404

http://kerneltrap.org/node/8206


Regards

Kishore


On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan <kernel.vijay@gmail.com>wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120621/9cd13cf5/attachment.html 

Kernel Memory

[Arun KS]

Hello Vijay,

On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan <kernel.vijay@gmail.com> wrote:
> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
You might have heard about 3G/1G split. This 1GB is the virtual
address of the kernel. And whenever kernel try to access any address
in this range(for ARM architecture it is  0xC0000000 - 0xFFFFFFFF),
there should not be any page fault. That means MMU should be able to
convert your virtual address to physical. This 1GB contains your IO
address, RAM address.

Paging is a mechanism OS uses to pull the data(in pagesizes) to and
fro between system RAM and secondary memory.
Kernel memory is not pageable. This means memory allocated for the
kernel will not be pagged out. If you try to access any memory in
kernel with out creating page tables(this can be done by ioremap) you
will end up in OOPS. The main reason of kernel not being swapable or
pageable is as follows.
Think this way. What will happen if we have paged out that portion of
the logic which decides what to do when a page fault occurs? Who will
take care of the page fault then?

But if a user program hit a page fault(ie accessed address is not in
main memory), kernel will load the page from secondary memory if it is
a valid address. And if the address accesses is illegal, kernel kill
the user application(Segmentation fault).

Thanks,
Arun

>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel Memory

[AFT]

Vijay Chauhan <kernel.vijay@gmail.com> writes:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>

Yes. Kernel works on static adress space.

> Any simple explanation will help me to udnerstand.
>

I'm not sure if you want to understand "how kernel manages memory for
its internal DS". If its the case you should read the following
documents.

1) Read the chapter 8 of Linux Device driver 3rd edition.

http://lwn.net/Kernel/LDD3/

2) To understand slab allocator read the following papers by bonwick

   a) 94 paper describing slab allocator:
   
   http://static.usenix.org/publications/library/proceedings/bos94/full_papers/bonwick.a

   b) Its followup in 2001
   
   http://static.usenix.org/event/usenix01/full_papers/bonwick/bonwick_html/index.html

3) These should be enough. But if you want to know detailed architecture
of how virtual memory manager work you should read Gorman's book on
Linux virtual memory manager. Its a free pdf. can be found here:

http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf

Happy hacking.

Cheers
aft


> Thanks,
> Vijay
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel Memory

[michi1 at michaelblizek.twilightparadox.com]

Hi!

On 17:57 Thu 21 Jun     , Vijay Chauhan wrote:
> Hello,
> 
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
> 
> Any simple explanation will help me to udnerstand.

The right term is actually "kernel memory is not swappable". Swapping means
writing inactive memory to disk and then using it for something else. Kernel
memory not being swappable is a design decicion made in the early linux days.
Operating systems which swap kernel memory need to isolate everything which
should not be swappd out (e.g. things needed for swap-in, realtime stuff,
security sensitive data, ...). This is quite a bit of work. I also guess it is
pretty pointless nowadays. Installed memory and is getting so huge that virtual
memory developers have a hard time trying to keep cpu-usage overhead for
swapping user space memory low.

> There is no concept of kernel virtual address?

Kernel memory uses virtual addresses as well. However, these the entire system
memory is continuously mapped somewhere in the virtual address space. The
drawback is that fragmentation turns allocation of large continuous memory
regions into a game of luck.

There is also an virtual address area (vmalloc) which is used to dynamically
map multiple scattered pages to a continuous region. But this is rather slow
and rarely used.

You might want to take a look at: http://lwn.net/Kernel/LDD3/

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Kernel Memory

[Vijay Chauhan]

Thanks everyone. I think i got enough information for further study.

Thanks,
Vijay

On Thu, Jun 21, 2012 at 7:10 PM, kishore sheik ahamed
<linuxinme@gmail.com> wrote:
> Hey Vijay
>
> I am a newbie too. Just sharing what I could go through.
>
> It is said that Kernel or atleast a part of kernel needs to be non paged for
> fast interrupt access etc as pinned memory
> Wiki says
>
> Pinned/Locked/Fixed pages
>
> Operating systems have memory areas that are pinned (never swapped to
> secondary storage). For example, interrupt mechanisms rely on an array of
> pointers to their handlers, such as I/O completion and page fault. If the
> pages containing these pointers or the code that they invoke were pageable,
> interrupt-handling would become far more complex and time-consuming,
> particularly in the case of page fault interrupts. Hence, some part of the
> page table structures is not pageable.
>
> Some pages may be pinned for short periods of time, others may be pinned for
> long periods of time, and still others may need to be permanently pinned.
> For example:
>
> The paging supervisor code and drivers for secondary storage devices on
> which pages reside must be permanently pinned, as otherwise paging wouldn't
> even work because the necessary code wouldn't be available.
> Timing-dependent components may be pinned to avoid variable paging delays.
> Data buffers that are accessed directly by peripheral devices that use
> direct memory access or I/O channels must reside in pinned pages while the
> I/O operation is in progress because such devices and the buses to which
> they are attached expect to find data buffers located at physical memory
> addresses; regardless of whether the bus has a memory management unit for
> I/O, transfers cannot be stopped if a page fault occurs and then restarted
> when the page fault has been processed.
>
> There are other two discussion thread which say kernel is non-pageable and
> now due to growing kernel Data structures it is allowed
>
> http://kerneltrap.org/node/6404
>
> http://kerneltrap.org/node/8206
>
>
> Regards
>
> Kishore
>
>
>
> On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan <kernel.vijay@gmail.com>
> wrote:
>>
>> Hello,
>>
>> I am newbie.
>> It has been said "kernel memory is not pageable"
>> What does it mean? There is no concept of kernel virtual address?
>>
>> Any simple explanation will help me to udnerstand.
>>
>> Thanks,
>> Vijay
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

Kernel memory

[Ronit Halder]

Hi,

Where in the memory kernel is located in the boot time?

regards,
Ronit

Kernel memory

[Valdis.Kletnieks at vt.edu]

On Tue, 17 May 2016 12:41:44 +0530, Ronit Halder said:
> Hi,
>
> Where in the memory kernel is located in the boot time?

During which exact phase of the boot, and does it actually matter?  And
physical or virtual address?

(Hint:  If you're not the bootstrap that unpacks the compressed kernel
into memory, or the code that sets up the ASLR code, or are trying to work
around an issue with a hardware device that insists on taking certain
physical addresses for itself, you probably don't really care where the
kernel is....)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160517/a4da7bca/attachment.bin