Bootstrap in mini-os

Bootstrap in mini-os

[Jayaraman, Bhaskar]

Hi i hope this is the right mailing list for discussing stuff on Mini-OS.

1] I have been looking through its sources and I wanted to know what is the bootstrap code comprised of in Mini-os.

2] The start_info pages seem to contain bootstrap pages in them and also the mini-os kernel text loads itself at virtual address 0x00 as directed by the lds script so does the bootstrap code come from Xen loader.

3] Also I guess the start pfn begins after the bootstrap pfn as mentioned here??

 /* First page follows page table pages and 3 more pages (store page etc) */
    start_pfn = PFN_UP(to_phys(start_info.pt_base)) +
                start_info.nr_pt_frames + 3;

If so why is the text virtual address being deducted from bootstrap page in to_phys ??

#define VIRT_START                 ((unsigned long)&_text)
#define to_phys(x)                 ((unsigned long)(x)-VIRT_START)

4] So kernel text will reside before bootstrap code? Which is why I'm curious to know if this is part of bootloader code.

5] What is being achieved by deducting the kernel text virtual address from the bootstrap virtual address? I mean why do we not want to account for that space while organizing mini-os memory?
i.e. if text were to be loaded at 1MB and bootstrap was at 2 MB we will have start pfn at 1 MB according to the calculations above and not at 2MB. Why is that so?

6] Why are we Oring 7 with shared info page while mapping it with the hypervisor?
HYPERVISOR_update_va_mapping(
                (unsigned long)shared_info, __pte(pa | 7), UVMF_INVLPG)


7] Will this virtual address be listed in the mfn_list, the page frame list for this VM before this hypercall?

Regards,
Bhaskar.

Re: Bootstrap in mini-os

[Samuel Thibault]

Jayaraman, Bhaskar, le Fri 27 Jun 2008 02:55:58 +0800, a écrit :
> 1] I have been looking through its sources and I wanted to know what is the bootstrap code comprised of in Mini-os.

Things start from arch/x86/x86_32.S:_start, then see function
start_kernel.

> 2] The start_info pages seem to contain bootstrap pages in them and also the mini-os kernel text loads itself at virtual address 0x00 as directed by the lds script so does the bootstrap code come from Xen loader.

No, everything is contained in the mini-os source. All Xen does is
mapping the ELF at the virtual address given in the lds, and setup a
bootstrap page table.

> 3] Also I guess the start pfn begins after the bootstrap pfn as mentioned here??

See the documentation about day0 in start_info doc in xen/include/public/xen.h
In mini-os, what is called "start_pfn" is just the first pfn which is
available for data/heap/whatever.

>  /* First page follows page table pages and 3 more pages (store page etc) */
>     start_pfn = PFN_UP(to_phys(start_info.pt_base)) +
>                 start_info.nr_pt_frames + 3;
> 
> If so why is the text virtual address being deducted from bootstrap page in to_phys ??

Because what is calculated above is not anything virtual, it's a PFN,
which are always numbered from 0, and thus you need to subtract the
VIRT_START offset.

> 4] So kernel text will reside before bootstrap code?

bootstrap code is _in_ the kernel. Have a look at an objdump -D of mini-os.

> Which is why I'm curious to know if this is part of bootloader code.

It is not.

> 5] What is being achieved by deducting the kernel text virtual address from the bootstrap virtual address?

What the macro says: going down from virtual address to physical
address.

> I mean why do we not want to account for that space while organizing mini-os memory?

Because it's easier to think of pages as starting from 0, wherever they
are virtually mapped.

> 6] Why are we Oring 7 with shared info page while mapping it with the hypervisor?
> HYPERVISOR_update_va_mapping(
>                 (unsigned long)shared_info, __pte(pa | 7), UVMF_INVLPG)

7 == _PAGE_PRESENT | _PAGE_RW | _PAGE_USER.

I guess a cleanup patch would indeed consist in replacing 7 with the OR
above.

> 7] Will this virtual address be listed in the mfn_list, the page frame list for this VM before this hypercall?

mfn_list does not reference virtual addresses, it lists MFNs. See the
comment in x86_32.S:

/* Unpleasant -- the PTE that maps this page is actually overwritten */
/* to map the real shared-info page! :-)     

The MFN which was previously mapped at that place will still be
referenced in the mfn_list and Xen will still consider it to be
allocated to Mini-OS. Mini-OS could very well project it somewhere else,
but it doesn't for simplicity (not a big loss).

Samuel

RE: Bootstrap in mini-os

[Jayaraman, Bhaskar]

Thanks Sam that has been quite insightful, I'm studying the feasibility of using Mini-OS for parallelizing I/Os on our platform actually. Looking into the objdump certainly helped. It'll be very helpful if you could clear a few more things up:-

1] Since the bootstrap page tables are marked read-only we have our start_pfn after it?

2] The domain loader will load the Mini-OS at pseudo physical page 0?

3] I didn't understand how does deducting &_text give a physical address, since the start pfn is to begin after pt_base and not at pseudo physical address 0.
Plus even if the text was loaded at any other place than pseudo pfn 0, the start pfn will still not be from 0.
What I'm saying is what difference will it make if we don't deduct the text pseudo physical address? i.e. as you mentioned "it's easier to think of pages as starting from 0" but start_pfn doesn't seem to be 0 based on the calculation.

4] If I'm right the shared_info pte is overwritten by the hypervisor with the shared_info struct that we pass it and relocated to another mfn in the mfn_list with the hypercall? However, Mini-os will still keep referring to shared_info at pseudo physical page at offset 0x1000 or the 1st pseudo physical frame?

Regards,
Bhaskar.


-----Original Message-----
From: Samuel Thibault [mailto:samuel.thibault@eu.citrix.com]
Sent: Friday, June 27, 2008 4:20 AM
To: Jayaraman, Bhaskar
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Bootstrap in mini-os

Jayaraman, Bhaskar, le Fri 27 Jun 2008 02:55:58 +0800, a écrit :
> 1] I have been looking through its sources and I wanted to know what is the bootstrap code comprised of in Mini-os.

Things start from arch/x86/x86_32.S:_start, then see function
start_kernel.

> 2] The start_info pages seem to contain bootstrap pages in them and also the mini-os kernel text loads itself at virtual address 0x00 as directed by the lds script so does the bootstrap code come from Xen loader.

No, everything is contained in the mini-os source. All Xen does is
mapping the ELF at the virtual address given in the lds, and setup a
bootstrap page table.

> 3] Also I guess the start pfn begins after the bootstrap pfn as mentioned here??

See the documentation about day0 in start_info doc in xen/include/public/xen.h
In mini-os, what is called "start_pfn" is just the first pfn which is
available for data/heap/whatever.

>  /* First page follows page table pages and 3 more pages (store page etc) */
>     start_pfn = PFN_UP(to_phys(start_info.pt_base)) +
>                 start_info.nr_pt_frames + 3;
>
> If so why is the text virtual address being deducted from bootstrap page in to_phys ??

Because what is calculated above is not anything virtual, it's a PFN,
which are always numbered from 0, and thus you need to subtract the
VIRT_START offset.

> 4] So kernel text will reside before bootstrap code?

bootstrap code is _in_ the kernel. Have a look at an objdump -D of mini-os.

> Which is why I'm curious to know if this is part of bootloader code.

It is not.

> 5] What is being achieved by deducting the kernel text virtual address from the bootstrap virtual address?

What the macro says: going down from virtual address to physical
address.

> I mean why do we not want to account for that space while organizing mini-os memory?

Because it's easier to think of pages as starting from 0, wherever they
are virtually mapped.

> 6] Why are we Oring 7 with shared info page while mapping it with the hypervisor?
> HYPERVISOR_update_va_mapping(
>                 (unsigned long)shared_info, __pte(pa | 7), UVMF_INVLPG)

7 == _PAGE_PRESENT | _PAGE_RW | _PAGE_USER.

I guess a cleanup patch would indeed consist in replacing 7 with the OR
above.

> 7] Will this virtual address be listed in the mfn_list, the page frame list for this VM before this hypercall?

mfn_list does not reference virtual addresses, it lists MFNs. See the
comment in x86_32.S:

/* Unpleasant -- the PTE that maps this page is actually overwritten */
/* to map the real shared-info page! :-)

The MFN which was previously mapped at that place will still be
referenced in the mfn_list and Xen will still consider it to be
allocated to Mini-OS. Mini-OS could very well project it somewhere else,
but it doesn't for simplicity (not a big loss).

Samuel

Re: Bootstrap in mini-os

[Samuel Thibault]

Jayaraman, Bhaskar, le Fri 27 Jun 2008 20:50:24 +0800, a écrit :
> 1] Since the bootstrap page tables are marked read-only we have our start_pfn after it?

Yes.

> 2] The domain loader will load the Mini-OS at pseudo physical page 0?

All PV kernels are always loaded at PFN 0. Then PFNs are mapped at the
virtual address set in the ELF notes.

> 3] I didn't understand how does deducting &_text give a physical address,

Because _text is always at PFN 0.

> since the start pfn is to begin after pt_base and not at pseudo physical address 0.

_text is not at start_pfn. start_pfn is _after_ all the day0 layout,
i.e. the kernel code, the bootstrap page table etc.

> What I'm saying is what difference will it make if we don't deduct the text pseudo physical address?

It does make a difference if somebody, for some reason, wants to have
the mini-os kernel virtually start at e.g. 0xc0000000. It will still
pseudo-physically start at 0.

> 4] If I'm right the shared_info pte is overwritten by the hypervisor with the shared_info struct that we pass it and relocated to another mfn in the mfn_list with the hypercall? However, Mini-os will still keep referring to shared_info at pseudo physical page at offset 0x1000 or the 1st pseudo physical frame?

Currently the PFN numbered 1 is not available from mini-os, because it
is "overmapped" by the shared info page.

Samuel