Query regarding Copy-on-write

Query regarding Copy-on-write

[Jagadeesh Bhaskar P]

Hi,

 When a process forks, every resource of the parent, including the
virtual memory is copied to the child process. The copying of VM uses
copy-on-write(COW). I know that COW comes when a write request comes,
and then the copy is made. Now my query follows:

How will the copy be distributed. Whether giving the child process a new
copy of VM be permanent or whether they will be merged anywhere? And
shouldn't the operations/updations by one process be visible to the
other which inherited the copy of the same VM?

How can this work? Can someone please help me on this regard?


-- 
With regards,

Jagadeesh Bhaskar P
R&D Engineer
HCL Infosystems Ltd
Pondicherry
INDIA

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Simon Valiquette]

Jagadeesh Bhaskar P a écrit :
> Hi,
> 
>  When a process forks, every resource of the parent, including the
> virtual memory is copied to the child process. The copying of VM uses
> copy-on-write(COW). I know that COW comes when a write request comes,
> and then the copy is made. Now my query follows:
> 
> How will the copy be distributed.

   If a child modify a page that is shared with his parent, this page 
will be copied to a new page.  Then his page table will be modified to 
point to this new page and reloaded in the MMU.  Only then the running 
process will normally modify his own copy of that page.

   It works the same way if it is the parent that modify the page. 
Otherwise we would have a problem if the parent have many childs. It 
also simplify things a lot for SMP systems.


   Be carefull: "page" and "page table" are 2 very different things.

   "Page table" will point to a page struct
   "page" is a struct that point to a physical page, or to a swap area 
(if I remember well).
   Virtual memory is the address seen by the program.  Two identical 
virtual adresses will normally point to different physical locations.


   Note that each process have his own copy of his memory page table 
since the fork.  And the page table will typically point to many memory 
pages.

   The shared pages pointed by the 2 page tables are only thoses where 
the data are still unmodified since the fork.


 > Whether giving the child process a new
 > copy of VM be permanent or whether they will be merged anywhere?

   Its permanent at the page level.  Anyway, once modified, it is very 
unlikely that the page will ever be identical again.

   Also, they never really shared the same virtual memory space since 
they each have their own page tables.  Their pointers use the same 
virtual addresses, but they are resolved to the same physical pages only 
for the pages that are still identicals.


   I'm not sure if you understand properly how the virtual memory is 
working at the processor level, but it is very important if you want to 
understand how COW is working.

> And
> shouldn't the operations/updations by one process be visible to the
> other which inherited the copy of the same VM?
> 

   No, it must be completelly transparent for the process and they 
should not be able to see the modifications (otherwise we have a 
security bug).

   You must know that the processor use the MMU to resolve the memory 
page physical address.  Since each process have his own memory table, a 
pointer to an identical virtual adress usually point to a different 
physical address.  So it means that once a shared page is copied and 
unlinked by modifying the page table of one of the process, the same 
pointer with the same pointing adress will now point to 2 different 
physical memory location.  All of that magic is done thanks to the MMU.


   I don't know if the kernel is doing COW at all in the case of a 
MMU-less processor (usually a microcontroler).

> How can this work? Can someone please help me on this regard?
> 

   I think I gave a good overview of the question.  I tried to keep 
things as simple as I could, at the cost of not giving an exact 
description of how Linux handle COW.  For example, there is in fact 3 
levels of page table indirections in Linux, but it was not necessary to 
understand COW.


Simon Valiquette

---
Often, the source contains important documentation
  - Slackware FAQ, 1993


-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Jim Nelson]

Jagadeesh Bhaskar P wrote:
> Hi,
> 
>  When a process forks, every resource of the parent, including the
> virtual memory is copied to the child process. The copying of VM uses
> copy-on-write(COW). I know that COW comes when a write request comes,
> and then the copy is made. Now my query follows:
> 
> How will the copy be distributed. Whether giving the child process a new
> copy of VM be permanent or whether they will be merged anywhere? And
> shouldn't the operations/updations by one process be visible to the
> other which inherited the copy of the same VM?
> 
> How can this work? Can someone please help me on this regard?
> 
> 

Standard *nix forking does not allow that kind of interaction between the data of 
parent and child processes.  Pipes are designed for that kind of interprocess 
communication.

POSIX threads do run in the same address space - it requires a bit more 
programming effort, but it seems to fit what you're asking for a little better.

http://www.cs.nmsu.edu/~jcook/Tools/pthreads/pthreads.html

Is a decent overview on pthreads.  I haven't used them - nothing I've written 
needed that kind of interactivity.

Not every computer system can run pthreads on Linux - i686 or higher, SPARC, and 
SPARC64 are the only ones I *know* have pthreads working (i. e. I own the machine 
and pthreaded Linux apps work on them).  i586 and earlier PC's lack some 
fundamental capabilities necessary to make threading work.

Anyone else care to take it from here?

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Jim Nelson]

Simon Valiquette wrote:
> Jagadeesh Bhaskar P a écrit :

>> How can this work? Can someone please help me on this regard?
>>
> 
>   I think I gave a good overview of the question.  I tried to keep 
> things as simple as I could, at the cost of not giving an exact 
> description of how Linux handle COW.  For example, there is in fact 3 
> levels of page table indirections in Linux, but it was not necessary to 
> understand COW.
> 

4-level page tables just were merged into the kernel mainline as of 2.6.10-rc2 - 
mostly to support the big iron that has almost-terabyte-level system memory (the 
SGI Altix, some of the big stuff from IBM, some other NUMA machines).

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Jagadeesh Bhaskar P]

Hi again,
  I got the concept. I had no problem understanding the explanation...I
already knew the concepts of virtual memory and paging so that was not a
problem.

  According to ur reply, after the sharing, will always the range of
addresses generated by the process, ie the virtual addresses be the same
of the parent, even if the physical memory pointed to by that is
different.

  So basically, is it that sharing VM imply that the range of addresses
is shared, and physical pages pointed to by them may or maynot be the
same?

Please do help, and thanks for the previous elaborate explanation!

On Sat, 2004-11-20 at 19:29, Simon Valiquette wrote:
> Jagadeesh Bhaskar P a écrit :
> > Hi,
> > 
> >  When a process forks, every resource of the parent, including the
> > virtual memory is copied to the child process. The copying of VM uses
> > copy-on-write(COW). I know that COW comes when a write request comes,
> > and then the copy is made. Now my query follows:
> > 
> > How will the copy be distributed.
> 
>    If a child modify a page that is shared with his parent, this page 
> will be copied to a new page.  Then his page table will be modified to 
> point to this new page and reloaded in the MMU.  Only then the running 
> process will normally modify his own copy of that page.
> 
>    It works the same way if it is the parent that modify the page. 
> Otherwise we would have a problem if the parent have many childs. It 
> also simplify things a lot for SMP systems.
> 
> 
>    Be carefull: "page" and "page table" are 2 very different things.
> 
>    "Page table" will point to a page struct
>    "page" is a struct that point to a physical page, or to a swap area 
> (if I remember well).
>    Virtual memory is the address seen by the program.  Two identical 
> virtual adresses will normally point to different physical locations.
> 
> 
>    Note that each process have his own copy of his memory page table 
> since the fork.  And the page table will typically point to many memory 
> pages.
> 
>    The shared pages pointed by the 2 page tables are only thoses where 
> the data are still unmodified since the fork.
> 
> 
>  > Whether giving the child process a new
>  > copy of VM be permanent or whether they will be merged anywhere?
> 
>    Its permanent at the page level.  Anyway, once modified, it is very 
> unlikely that the page will ever be identical again.
> 
>    Also, they never really shared the same virtual memory space since 
> they each have their own page tables.  Their pointers use the same 
> virtual addresses, but they are resolved to the same physical pages only 
> for the pages that are still identicals.
> 
> 
>    I'm not sure if you understand properly how the virtual memory is 
> working at the processor level, but it is very important if you want to 
> understand how COW is working.
> 
> > And
> > shouldn't the operations/updations by one process be visible to the
> > other which inherited the copy of the same VM?
> > 
> 
>    No, it must be completelly transparent for the process and they 
> should not be able to see the modifications (otherwise we have a 
> security bug).
> 
>    You must know that the processor use the MMU to resolve the memory 
> page physical address.  Since each process have his own memory table, a 
> pointer to an identical virtual adress usually point to a different 
> physical address.  So it means that once a shared page is copied and 
> unlinked by modifying the page table of one of the process, the same 
> pointer with the same pointing adress will now point to 2 different 
> physical memory location.  All of that magic is done thanks to the MMU.
> 
> 
>    I don't know if the kernel is doing COW at all in the case of a 
> MMU-less processor (usually a microcontroler).
> 
> > How can this work? Can someone please help me on this regard?
> > 
> 
>    I think I gave a good overview of the question.  I tried to keep 
> things as simple as I could, at the cost of not giving an exact 
> description of how Linux handle COW.  For example, there is in fact 3 
> levels of page table indirections in Linux, but it was not necessary to 
> understand COW.
> 
> 
> Simon Valiquette
> 
> ---
> Often, the source contains important documentation
>   - Slackware FAQ, 1993
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.linux-learn.org/faqs
-- 
With regards,

Jagadeesh Bhaskar P
R&D Engineer
HCL Infosystems Ltd
Pondicherry
INDIA

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Simon Valiquette]

Jagadeesh Bhaskar P a écrit :
> Hi again,
>   I got the concept. I had no problem understanding the explanation...I
> already knew the concepts of virtual memory and paging so that was not a
> problem.
> 
>   According to ur reply, after the sharing, will always the range of
> addresses generated by the process, ie the virtual addresses be the same
> of the parent, even if the physical memory pointed to by that is
> different.
> 

   Yes, otherwise the pointers would point to the wrong adresses and the 
program will probably quickly segfault.

>   So basically, is it that sharing VM imply that the range of addresses
> is shared, and physical pages pointed to by them may or maynot be the
> same?
> 

   The range of virtual addresses is reused, not shared, but I guess it 
is just a question of using the good words (you are probably also a 
non-native english speakers).  But yes that's the basic idea.

> Please do help, and thanks for the previous elaborate explanation!
> 

   When I first looked inside the Linux kernel, I started by trying to 
understand how it handled memory.  I finally realized how crazy I was, 
and switched to an easier part.  But by the time, I had learned quite a 
lot about VM, and also made a minimalist OS for a MC68000 microcontroler 
(only basic process support without even support for a filesystem). 
That's also probably why I don't think the actual kernel support 
copy-on-write on MMU-less processor, as I can't think of any good and 
efficient implementation for that.


Simon Valiquette

---
The memory management on the PowerPC can be used
to frighten small children.
   - Linus Torvalds

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Jagadeesh Bhaskar P]

On Mon, 2004-11-22 at 12:01, Simon Valiquette wrote:

>    The range of virtual addresses is reused, not shared, but I guess it 
> is just a question of using the good words (you are probably also a 
> non-native english speakers).  But yes that's the basic idea.
Yes, Im a non-native English speaker, and I got ur idea well and clear.
Thanks!!


> and also made a minimalist OS for a MC68000 microcontroler 
> (only basic process support without even support for a filesystem). 


I too want to learn something like a minimalist OS. Where to start
digging?

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Simon Valiquette]

Jagadeesh Bhaskar P a écrit :
> 
>>and also made a minimalist OS for a MC68000 microcontroler 
>>(only basic process support without even support for a filesystem). 
> 
> 
> 
> I too want to learn something like a minimalist OS. Where to start
> digging?
> 

   I'm not too sure.  In my case, I started mixing assembly language and 
Pascal about 15 years ago while quite young.  It really helps :o)



   I know there is things likes Nachos, OSP or Minix (yes, that's the 
one used by Linus) for learning how to write operating systems.  I 
personnally prefer a real OS like Hurd, but if you have a teacher or 
someone to assist you, I guess they can be usefull.


   To write a real OS, you need to understant how your hardware is 
working and to have the documentation book to know how to program it and 
manage it's interruptions.  Beeing efficient in assembly language is a 
must, even if you do most of your coding in C language, as it helps you 
to understand your computer.

   A book that might help you saving time, and that I found both very 
informative and interesting is this one:

   "Operating systems" from William Stallings, Prentice Hall.

   It won't tell you how to code everythings, but it will explain the 
algorithmes and ideas of differents Operating systems for doing similar 
tasks.  For example, it compares Posix threads with Linux, Windows and 
Solaris threads.  I found it quite complete; it even talk about RAID-6, 
and my book is many years old.


   Understanding as many OS as possible will certainly helps you 
understanding differents concepts and improve your skills.  OS likes 
Inferno, QNX and Plan 9 are probably good for that.


   For going back to your question, once you feel ready, you should 
probably start by trying to write a small OS.  At first, all you will 
need to write is a sheduler, a way to start programs and a way to kill 
them.  Writing something similar to /sbin/init with a couple of built-in 
commands (for starting, showing and stoping processes) will also 
simplify things.


   Eventually, you could add (or emulate) a file system.  By the way, 
the sheduler should be activated by interruptions, otherwise you will 
just have an improved version of DOS 1.0  ;o)


Simon Valiquette

---
win-nt from the people who invented edlin
(By mwikholm@at8.abo.fi, MaDsen Wikholm)

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

[Manish Regmi]

On Mon, 22 Nov 2004 06:48:00 -0500, Simon Valiquette <v.simon@ieee.org> wrote:
> Jagadeesh Bhaskar P a écrit :
> >
> >>and also made a minimalist OS for a MC68000 microcontroler
> >>(only basic process support without even support for a filesystem).
> >
> >
> >
> > I too want to learn something like a minimalist OS. Where to start
> > digging?
> >
> 
>   I'm not too sure.  In my case, I started mixing assembly language and
> Pascal about 15 years ago while quite young.  It really helps :o)
> 
>   I know there is things likes Nachos, OSP or Minix (yes, that's the
> one used by Linus) for learning how to write operating systems.  I
> personnally prefer a real OS like Hurd, but if you have a teacher or
> someone to assist you, I guess they can be usefull.
> 
>   To write a real OS, you need to understant how your hardware is
> working and to have the documentation book to know how to program it and
> manage it's interruptions.  Beeing efficient in assembly language is a
> must, even if you do most of your coding in C language, as it helps you
> to understand your computer.
> 
>   A book that might help you saving time, and that I found both very
> informative and interesting is this one:
> 
>   "Operating systems" from William Stallings, Prentice Hall.
> 
>   It won't tell you how to code everythings, but it will explain the
> algorithmes and ideas of differents Operating systems for doing similar
> tasks.  For example, it compares Posix threads with Linux, Windows and
> Solaris threads.  I found it quite complete; it even talk about RAID-6,
> and my book is many years old.
> 
>   Understanding as many OS as possible will certainly helps you
> understanding differents concepts and improve your skills.  OS likes
> Inferno, QNX and Plan 9 are probably good for that.
> 
>   For going back to your question, once you feel ready, you should
> probably start by trying to write a small OS.  At first, all you will
> need to write is a sheduler, a way to start programs and a way to kill
> them.  Writing something similar to /sbin/init with a couple of built-in
> commands (for starting, showing and stoping processes) will also
> simplify things.
> 
>   Eventually, you could add (or emulate) a file system.  By the way,
> the sheduler should be activated by interruptions, otherwise you will
> just have an improved version of DOS 1.0  ;o)
> 
> Simon Valiquette
> 
> ---
> win-nt from the people who invented edlin
> (By mwikholm@at8.abo.fi, MaDsen Wikholm)



Furthermore,
 There are a lot of resources in the internet. I too started writing
an OS few years back. Those resources were very much helpful.

www.osdev.org
www.nondot.org/~sabre/os
www.mega-tokyo.org/forum
http://my.execpc.com/~geezer/os/
http://mega-tokyo.com/osfaq2/
sources of Linux, Mach, e.t.c.
(google)
http://www.onesmartclick.com/rtos/rtos.html (if you are concerned with rtos)

regards manish


-- 
Manish Regmi
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs