Linux Stubdom Problem

Linux Stubdom Problem

[Jiageng Yu]

2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> CC'ing Tim and xen-devel
>
> On Mon, 18 Jul 2011, Jiageng Yu wrote:
>> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
>> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
>> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
>> >> >>>
>> >> >>>
>> >> >>> The HVM guest is booted. But the boot process is terminated because
>> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
>> >> >>>
>> >> >>
>> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
>> >> >> being loaded, it should be fixed now.
>> >> >>
>> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
>> >> >>
>> >> >> ---
>> >> >>
>> >> >>
>> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
>> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
>> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
>> >> >> @@ -430,7 +430,7 @@ int main(void)
>> >> >>             bios->create_pir_tables();
>> >> >>     }
>> >> >>
>> >> >> -    if ( bios->load_roms )
>> >> >> +    if ( 1 )
>> >> >>     {
>> >> >>         switch ( virtual_vga )
>> >> >>         {
>> >> >>
>> >> >>
>> >> >
>> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
>> >> > signal subsequently. I am trying to print the call stack when
>> >> > receiving the signal.
>> >> >
>> >>
>> >> Hi,
>> >>
>> >>    I find the cause of SIGSEGV signal:
>> >>
>> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
>> >> len, int is_write)
>> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
>> >>
>> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
>> >> 0x53e address, the SIGSEGV signal is generated.
>> >>
>> >>     I believe the qemu is trying to vist vram in this moment. This
>> >> code seems no problem, and I will continue to find the root cause.
>> >>
>> >
>> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
>> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
>> > xc_domain_populate_physmap_exact.
>> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
>> > add the missing vram pages in the guest. Maybe this hypercall is failing
>> > in your case?
>>
>>
>> Hi,
>>
>>    I continue to invesgate this bug and find hypercall_mmu_update in
>> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
>>
>> do_mmu_update
>>       ->mod_l1_entry
>>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
>>                          return -EINVAL;
>>
>>    mfn==INVALID_MFN, because :
>>
>> mod_l1_entry
>>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
>>               ->p2m->get_entry
>>                         ->p2m_gfn_to_mfn
>>                                -> if ( gfn > p2m->max_mapped_pfn )
>>                                    /* This pfn is higher than the
>> highest the p2m map currently holds */
>>                                    return _mfn(INVALID_MFN);
>>
>>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
>> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
>> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
>> will exceed 0xfffff.
>>
>>    In the case of minios based stubdom, the mmu_update.vals do not
>> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
>> exceeds 0x8000000100000000.
>
> It looks like the address of the guest that qemu is trying to map is not
> valid.
> Make sure you are running a guest with less than 2GB of ram, otherwise
> you need the patch series that Anthony sent on Friday:
>
> http://marc.info/?l=qemu-devel&m=131074042905711&w=2

Not this problem. I never alloc more than 2GB for the hvm guest. The
call stack in qemu is:

qemu_get_ram_ptr
      ->qemu_map_cache(addr, 0, 1)
                 -> if (!entry->vaddr_base || entry->paddr_index !=
address_index ||
                                          !test_bit(address_offset >>
XC_PAGE_SHIFT, entry->valid_mapping)) {
                           qemu_remap_bucket(entry, size ? :
MCACHE_BUCKET_SIZE, address_index);
                                 ->xc_map_foreign_bulk(xen_xc,
xen_domid, PROT_READ|PROT_WRITE,

                pfns, err, nb_pfn);

The qemu tries to map pages from hvm guest(xen_domid) to linux
stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
= 0xfffff)

    if ( gfn > p2m->max_mapped_pfn )
        /* This pfn is higher than the highest the p2m map currently holds */
        return _mfn(INVALID_MFN);

 In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
Maybe the address translation in linux stubdom cause this probem?

 BTW, in minios stubdom case, there seems no hvmloader process. Is it
needed in linux stubdom?

 Thanks,

Jiageng Yu.

iommu=force-hpdl385g7 boot option as workaround for missing IOMMU support in BIOS of HP DL385 g7

[Mark Schneider]

Hello,

How difficult would be to extend xen 4.1 code to create missing IVRS 
tables for "broken BIOSes"?
(Like BIOS of HP DL385g7 and many other servers/mainboards)

Thank you in advance for any hints.

Regards
Mark Schneider

-- 
ms@it-infrastrukturen.org

+++ Free infrastructure software for all. +++

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5990 bytes --]

On Thu, 21 Jul 2011, Jiageng Yu wrote:
> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > CC'ing Tim and xen-devel
> >
> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
> >> >> >>>
> >> >> >>>
> >> >> >>> The HVM guest is booted. But the boot process is terminated because
> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
> >> >> >>>
> >> >> >>
> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
> >> >> >> being loaded, it should be fixed now.
> >> >> >>
> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
> >> >> >>
> >> >> >> ---
> >> >> >>
> >> >> >>
> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> >> >> >>             bios->create_pir_tables();
> >> >> >>     }
> >> >> >>
> >> >> >> -    if ( bios->load_roms )
> >> >> >> +    if ( 1 )
> >> >> >>     {
> >> >> >>         switch ( virtual_vga )
> >> >> >>         {
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
> >> >> > signal subsequently. I am trying to print the call stack when
> >> >> > receiving the signal.
> >> >> >
> >> >>
> >> >> Hi,
> >> >>
> >> >>    I find the cause of SIGSEGV signal:
> >> >>
> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
> >> >> len, int is_write)
> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
> >> >>
> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
> >> >> 0x53e address, the SIGSEGV signal is generated.
> >> >>
> >> >>     I believe the qemu is trying to vist vram in this moment. This
> >> >> code seems no problem, and I will continue to find the root cause.
> >> >>
> >> >
> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
> >> > xc_domain_populate_physmap_exact.
> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
> >> > in your case?
> >>
> >>
> >> Hi,
> >>
> >>    I continue to invesgate this bug and find hypercall_mmu_update in
> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> >>
> >> do_mmu_update
> >>       ->mod_l1_entry
> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
> >>                          return -EINVAL;
> >>
> >>    mfn==INVALID_MFN, because :
> >>
> >> mod_l1_entry
> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
> >>               ->p2m->get_entry
> >>                         ->p2m_gfn_to_mfn
> >>                                -> if ( gfn > p2m->max_mapped_pfn )
> >>                                    /* This pfn is higher than the
> >> highest the p2m map currently holds */
> >>                                    return _mfn(INVALID_MFN);
> >>
> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
> >> will exceed 0xfffff.
> >>
> >>    In the case of minios based stubdom, the mmu_update.vals do not
> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
> >> exceeds 0x8000000100000000.
> >
> > It looks like the address of the guest that qemu is trying to map is not
> > valid.
> > Make sure you are running a guest with less than 2GB of ram, otherwise
> > you need the patch series that Anthony sent on Friday:
> >
> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> 
> Not this problem. I never alloc more than 2GB for the hvm guest. The
> call stack in qemu is:
> 
> qemu_get_ram_ptr
>       ->qemu_map_cache(addr, 0, 1)
>                  -> if (!entry->vaddr_base || entry->paddr_index !=
> address_index ||
>                                           !test_bit(address_offset >>
> XC_PAGE_SHIFT, entry->valid_mapping)) {
>                            qemu_remap_bucket(entry, size ? :
> MCACHE_BUCKET_SIZE, address_index);
>                                  ->xc_map_foreign_bulk(xen_xc,
> xen_domid, PROT_READ|PROT_WRITE,
> 
>                 pfns, err, nb_pfn);
> 
> The qemu tries to map pages from hvm guest(xen_domid) to linux
> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
> = 0xfffff)
> 
>     if ( gfn > p2m->max_mapped_pfn )
>         /* This pfn is higher than the highest the p2m map currently holds */
>         return _mfn(INVALID_MFN);
> 
>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
> Maybe the address translation in linux stubdom cause this probem?

Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
does not exceed 2G:

0xfffff * 4096 > 2G


>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
> needed in linux stubdom?

hvmloader is the first thing that runs within the guest, it is not a
process in the stubdom or in dom0.
It is required in both minios and linux stubdoms.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Thu, 21 Jul 2011, Jiageng Yu wrote:
>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> > CC'ing Tim and xen-devel
>> >
>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
>> >> >> >>>
>> >> >> >>
>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
>> >> >> >> being loaded, it should be fixed now.
>> >> >> >>
>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
>> >> >> >>
>> >> >> >> ---
>> >> >> >>
>> >> >> >>
>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
>> >> >> >>             bios->create_pir_tables();
>> >> >> >>     }
>> >> >> >>
>> >> >> >> -    if ( bios->load_roms )
>> >> >> >> +    if ( 1 )
>> >> >> >>     {
>> >> >> >>         switch ( virtual_vga )
>> >> >> >>         {
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
>> >> >> > signal subsequently. I am trying to print the call stack when
>> >> >> > receiving the signal.
>> >> >> >
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >>    I find the cause of SIGSEGV signal:
>> >> >>
>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
>> >> >> len, int is_write)
>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
>> >> >>
>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
>> >> >> 0x53e address, the SIGSEGV signal is generated.
>> >> >>
>> >> >>     I believe the qemu is trying to vist vram in this moment. This
>> >> >> code seems no problem, and I will continue to find the root cause.
>> >> >>
>> >> >
>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
>> >> > xc_domain_populate_physmap_exact.
>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
>> >> > in your case?
>> >>
>> >>
>> >> Hi,
>> >>
>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
>> >>
>> >> do_mmu_update
>> >>       ->mod_l1_entry
>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
>> >>                          return -EINVAL;
>> >>
>> >>    mfn==INVALID_MFN, because :
>> >>
>> >> mod_l1_entry
>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
>> >>               ->p2m->get_entry
>> >>                         ->p2m_gfn_to_mfn
>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
>> >>                                    /* This pfn is higher than the
>> >> highest the p2m map currently holds */
>> >>                                    return _mfn(INVALID_MFN);
>> >>
>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
>> >> will exceed 0xfffff.
>> >>
>> >>    In the case of minios based stubdom, the mmu_update.vals do not
>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
>> >> exceeds 0x8000000100000000.
>> >
>> > It looks like the address of the guest that qemu is trying to map is not
>> > valid.
>> > Make sure you are running a guest with less than 2GB of ram, otherwise
>> > you need the patch series that Anthony sent on Friday:
>> >
>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
>>
>> Not this problem. I never alloc more than 2GB for the hvm guest. The
>> call stack in qemu is:
>>
>> qemu_get_ram_ptr
>>       ->qemu_map_cache(addr, 0, 1)
>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
>> address_index ||
>>                                           !test_bit(address_offset >>
>> XC_PAGE_SHIFT, entry->valid_mapping)) {
>>                            qemu_remap_bucket(entry, size ? :
>> MCACHE_BUCKET_SIZE, address_index);
>>                                  ->xc_map_foreign_bulk(xen_xc,
>> xen_domid, PROT_READ|PROT_WRITE,
>>
>>                 pfns, err, nb_pfn);
>>
>> The qemu tries to map pages from hvm guest(xen_domid) to linux
>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
>> = 0xfffff)
>>
>>     if ( gfn > p2m->max_mapped_pfn )
>>         /* This pfn is higher than the highest the p2m map currently holds */
>>         return _mfn(INVALID_MFN);
>>
>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
>> Maybe the address translation in linux stubdom cause this probem?
>
> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
> does not exceed 2G:
>
> 0xfffff * 4096 > 2G
>
>
>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
>> needed in linux stubdom?
>
> hvmloader is the first thing that runs within the guest, it is not a
> process in the stubdom or in dom0.
> It is required in both minios and linux stubdoms.

Hi Stefano,

      I patched these patches, but we still have the same problem.
However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
vga_common_init function was also failed. Maybe this can explain the
previous problem, which happened in the phase of trying to remap
0xc0000-0xc8fff of hvm guest into stubdom.

     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
located the failure in p2m_gfn_to_mfn function:

     pod_retry_l3:
        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
        {
                 .....
                 return _mfn(INVALID_MFN);
        }

     I will continue to analyze this failure.

     Thanks!

Jiageng Yu.

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> On Thu, 21 Jul 2011, Jiageng Yu wrote:
>>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>>> > CC'ing Tim and xen-devel
>>> >
>>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
>>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
>>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
>>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
>>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
>>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
>>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
>>> >> >> >>>
>>> >> >> >>
>>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
>>> >> >> >> being loaded, it should be fixed now.
>>> >> >> >>
>>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
>>> >> >> >>
>>> >> >> >> ---
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
>>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
>>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
>>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
>>> >> >> >>             bios->create_pir_tables();
>>> >> >> >>     }
>>> >> >> >>
>>> >> >> >> -    if ( bios->load_roms )
>>> >> >> >> +    if ( 1 )
>>> >> >> >>     {
>>> >> >> >>         switch ( virtual_vga )
>>> >> >> >>         {
>>> >> >> >>
>>> >> >> >>
>>> >> >> >
>>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
>>> >> >> > signal subsequently. I am trying to print the call stack when
>>> >> >> > receiving the signal.
>>> >> >> >
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >>    I find the cause of SIGSEGV signal:
>>> >> >>
>>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
>>> >> >> len, int is_write)
>>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
>>> >> >>
>>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
>>> >> >> 0x53e address, the SIGSEGV signal is generated.
>>> >> >>
>>> >> >>     I believe the qemu is trying to vist vram in this moment. This
>>> >> >> code seems no problem, and I will continue to find the root cause.
>>> >> >>
>>> >> >
>>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
>>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
>>> >> > xc_domain_populate_physmap_exact.
>>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
>>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
>>> >> > in your case?
>>> >>
>>> >>
>>> >> Hi,
>>> >>
>>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
>>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
>>> >>
>>> >> do_mmu_update
>>> >>       ->mod_l1_entry
>>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
>>> >>                          return -EINVAL;
>>> >>
>>> >>    mfn==INVALID_MFN, because :
>>> >>
>>> >> mod_l1_entry
>>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
>>> >>               ->p2m->get_entry
>>> >>                         ->p2m_gfn_to_mfn
>>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
>>> >>                                    /* This pfn is higher than the
>>> >> highest the p2m map currently holds */
>>> >>                                    return _mfn(INVALID_MFN);
>>> >>
>>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
>>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
>>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
>>> >> will exceed 0xfffff.
>>> >>
>>> >>    In the case of minios based stubdom, the mmu_update.vals do not
>>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
>>> >> exceeds 0x8000000100000000.
>>> >
>>> > It looks like the address of the guest that qemu is trying to map is not
>>> > valid.
>>> > Make sure you are running a guest with less than 2GB of ram, otherwise
>>> > you need the patch series that Anthony sent on Friday:
>>> >
>>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
>>>
>>> Not this problem. I never alloc more than 2GB for the hvm guest. The
>>> call stack in qemu is:
>>>
>>> qemu_get_ram_ptr
>>>       ->qemu_map_cache(addr, 0, 1)
>>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
>>> address_index ||
>>>                                           !test_bit(address_offset >>
>>> XC_PAGE_SHIFT, entry->valid_mapping)) {
>>>                            qemu_remap_bucket(entry, size ? :
>>> MCACHE_BUCKET_SIZE, address_index);
>>>                                  ->xc_map_foreign_bulk(xen_xc,
>>> xen_domid, PROT_READ|PROT_WRITE,
>>>
>>>                 pfns, err, nb_pfn);
>>>
>>> The qemu tries to map pages from hvm guest(xen_domid) to linux
>>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
>>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
>>> = 0xfffff)
>>>
>>>     if ( gfn > p2m->max_mapped_pfn )
>>>         /* This pfn is higher than the highest the p2m map currently holds */
>>>         return _mfn(INVALID_MFN);
>>>
>>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
>>> Maybe the address translation in linux stubdom cause this probem?
>>
>> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
>> does not exceed 2G:
>>
>> 0xfffff * 4096 > 2G
>>
>>
>>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
>>> needed in linux stubdom?
>>
>> hvmloader is the first thing that runs within the guest, it is not a
>> process in the stubdom or in dom0.
>> It is required in both minios and linux stubdoms.
>
> Hi Stefano,
>
>      I patched these patches, but we still have the same problem.
> However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> vga_common_init function was also failed. Maybe this can explain the
> previous problem, which happened in the phase of trying to remap
> 0xc0000-0xc8fff of hvm guest into stubdom.
>
>     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
> located the failure in p2m_gfn_to_mfn function:
>
>     pod_retry_l3:
>        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
>        {
>                 .....
>                 return _mfn(INVALID_MFN);
>        }
>
>     I will continue to analyze this failure.
>
>     Thanks!
>
> Jiageng Yu.
>


Hi,

    I compared the two executions of vga_common_init function in dom0
and linux based stubdom. The former succeeded and the later was
failed. First, they have the same call stack:

Dom0 & Stubdom
_________________________________________________________
vga_common_init
     ->qemu_get_ram_ptr(s->vram_offset)
           ->block->host = xen_map_block(block->offset, block->length);
                 ->xc_map_foreign_bulk()
                        ->linux_privcmd_map_foreign_bulk()
                               ->xen_remap_domain_mfn_range()
                                     ->HYPERVISOR_mmu_update()
__________________________________________________________

Xen
__________________________________________________________
do_mmu_update()
   ->case MMU_PT_UPDATE_PRESERVE_AD:
   ->case PGT_l1_page_table:
   ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v, pg_owner);
          ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
l1e_get_pfn(nl1e), &p2mt));
                 ->gfn_to_mfn_type_p2m()
                        ->p2m->get_entry(p2m, gfn, t, &a, q);
                               ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
                                      ->if ( (l3e_get_flags(*l3e) &
_PAGE_PRESENT) == 0 )
                                      ->    Error happens!

The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
address area(pfn:0x40000, size:8M) into linux stubdom's address space.

In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
gfn=0x40000, t=p2m_mmio_dm.
mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
map_domain_page(mfn_x(mfn)) is also success. However, after executing:
l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
the l3e->l3 =0 , and the error happens.

So, in linux stubdom, when we are going to map the specified hvm
guest's address(pfn:0x40000, size:8M), we find these pages of hvm
guest are not present. This is never happened in qemu of dom0. Could
you give me some prompts to this problem?

Thanks！

Jiageng Yu.

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 10225 bytes --]

On Tue, 26 Jul 2011, Jiageng Yu wrote:
> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >>> > CC'ing Tim and xen-devel
> >>> >
> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> >>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> >>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
> >>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
> >>> >> >> >>>
> >>> >> >> >>>
> >>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
> >>> >> >> >>>
> >>> >> >> >>
> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
> >>> >> >> >> being loaded, it should be fixed now.
> >>> >> >> >>
> >>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
> >>> >> >> >>
> >>> >> >> >> ---
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> >>> >> >> >>             bios->create_pir_tables();
> >>> >> >> >>     }
> >>> >> >> >>
> >>> >> >> >> -    if ( bios->load_roms )
> >>> >> >> >> +    if ( 1 )
> >>> >> >> >>     {
> >>> >> >> >>         switch ( virtual_vga )
> >>> >> >> >>         {
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >
> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
> >>> >> >> > signal subsequently. I am trying to print the call stack when
> >>> >> >> > receiving the signal.
> >>> >> >> >
> >>> >> >>
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >>    I find the cause of SIGSEGV signal:
> >>> >> >>
> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
> >>> >> >> len, int is_write)
> >>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
> >>> >> >>
> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
> >>> >> >>
> >>> >> >>     I believe the qemu is trying to vist vram in this moment. This
> >>> >> >> code seems no problem, and I will continue to find the root cause.
> >>> >> >>
> >>> >> >
> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
> >>> >> > xc_domain_populate_physmap_exact.
> >>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
> >>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
> >>> >> > in your case?
> >>> >>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> >>> >>
> >>> >> do_mmu_update
> >>> >>       ->mod_l1_entry
> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
> >>> >>                          return -EINVAL;
> >>> >>
> >>> >>    mfn==INVALID_MFN, because :
> >>> >>
> >>> >> mod_l1_entry
> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
> >>> >>               ->p2m->get_entry
> >>> >>                         ->p2m_gfn_to_mfn
> >>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
> >>> >>                                    /* This pfn is higher than the
> >>> >> highest the p2m map currently holds */
> >>> >>                                    return _mfn(INVALID_MFN);
> >>> >>
> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
> >>> >> will exceed 0xfffff.
> >>> >>
> >>> >>    In the case of minios based stubdom, the mmu_update.vals do not
> >>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
> >>> >> exceeds 0x8000000100000000.
> >>> >
> >>> > It looks like the address of the guest that qemu is trying to map is not
> >>> > valid.
> >>> > Make sure you are running a guest with less than 2GB of ram, otherwise
> >>> > you need the patch series that Anthony sent on Friday:
> >>> >
> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> >>>
> >>> Not this problem. I never alloc more than 2GB for the hvm guest. The
> >>> call stack in qemu is:
> >>>
> >>> qemu_get_ram_ptr
> >>>       ->qemu_map_cache(addr, 0, 1)
> >>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
> >>> address_index ||
> >>>                                           !test_bit(address_offset >>
> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
> >>>                            qemu_remap_bucket(entry, size ? :
> >>> MCACHE_BUCKET_SIZE, address_index);
> >>>                                  ->xc_map_foreign_bulk(xen_xc,
> >>> xen_domid, PROT_READ|PROT_WRITE,
> >>>
> >>>                 pfns, err, nb_pfn);
> >>>
> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
> >>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
> >>> = 0xfffff)
> >>>
> >>>     if ( gfn > p2m->max_mapped_pfn )
> >>>         /* This pfn is higher than the highest the p2m map currently holds */
> >>>         return _mfn(INVALID_MFN);
> >>>
> >>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
> >>> Maybe the address translation in linux stubdom cause this probem?
> >>
> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
> >> does not exceed 2G:
> >>
> >> 0xfffff * 4096 > 2G
> >>
> >>
> >>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
> >>> needed in linux stubdom?
> >>
> >> hvmloader is the first thing that runs within the guest, it is not a
> >> process in the stubdom or in dom0.
> >> It is required in both minios and linux stubdoms.
> >
> > Hi Stefano,
> >
> >      I patched these patches, but we still have the same problem.
> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> > vga_common_init function was also failed. Maybe this can explain the
> > previous problem, which happened in the phase of trying to remap
> > 0xc0000-0xc8fff of hvm guest into stubdom.
> >
> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
> > located the failure in p2m_gfn_to_mfn function:
> >
> >     pod_retry_l3:
> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
> >        {
> >                 .....
> >                 return _mfn(INVALID_MFN);
> >        }
> >
> >     I will continue to analyze this failure.
> >
> >     Thanks!
> >
> > Jiageng Yu.
> >
> 
> 
> Hi,
> 
>     I compared the two executions of vga_common_init function in dom0
> and linux based stubdom. The former succeeded and the later was
> failed. First, they have the same call stack:
> 
> Dom0 & Stubdom
> _________________________________________________________
> vga_common_init
>      ->qemu_get_ram_ptr(s->vram_offset)
>            ->block->host = xen_map_block(block->offset, block->length);
>                  ->xc_map_foreign_bulk()
>                         ->linux_privcmd_map_foreign_bulk()
>                                ->xen_remap_domain_mfn_range()
>                                      ->HYPERVISOR_mmu_update()
> __________________________________________________________
> 
> Xen
> __________________________________________________________
> do_mmu_update()
>    ->case MMU_PT_UPDATE_PRESERVE_AD:
>    ->case PGT_l1_page_table:
>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v, pg_owner);
>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> l1e_get_pfn(nl1e), &p2mt));
>                  ->gfn_to_mfn_type_p2m()
>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
>                                       ->if ( (l3e_get_flags(*l3e) &
> _PAGE_PRESENT) == 0 )
>                                       ->    Error happens!
> 
> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
> address area(pfn:0x40000, size:8M) into linux stubdom's address space.
> 
> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
> gfn=0x40000, t=p2m_mmio_dm.
> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
> map_domain_page(mfn_x(mfn)) is also success. However, after executing:
> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
> the l3e->l3 =0 , and the error happens.
> 
> So, in linux stubdom, when we are going to map the specified hvm
> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
> guest are not present. This is never happened in qemu of dom0. Could
> you give me some prompts to this problem?


It seems that you are trying to map pages that don't exist.
The pages in question should be allocated by:

qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
    qemu_ram_alloc_from_ptr
        xen_ram_alloc
            xc_domain_populate_physmap_exact

so I would add some printf and printk on this code path to find out if
xc_domain_populate_physmap_exact fails for some reasons.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Tue, 26 Jul 2011, Jiageng Yu wrote:
>> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
>> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
>> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >>> > CC'ing Tim and xen-devel
>> >>> >
>> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
>> >>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
>> >>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> >>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
>> >>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
>> >>> >> >> >>>
>> >>> >> >> >>>
>> >>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
>> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
>> >>> >> >> >>>
>> >>> >> >> >>
>> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
>> >>> >> >> >> being loaded, it should be fixed now.
>> >>> >> >> >>
>> >>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
>> >>> >> >> >>
>> >>> >> >> >> ---
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
>> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
>> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
>> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
>> >>> >> >> >>             bios->create_pir_tables();
>> >>> >> >> >>     }
>> >>> >> >> >>
>> >>> >> >> >> -    if ( bios->load_roms )
>> >>> >> >> >> +    if ( 1 )
>> >>> >> >> >>     {
>> >>> >> >> >>         switch ( virtual_vga )
>> >>> >> >> >>         {
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >
>> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
>> >>> >> >> > signal subsequently. I am trying to print the call stack when
>> >>> >> >> > receiving the signal.
>> >>> >> >> >
>> >>> >> >>
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >>    I find the cause of SIGSEGV signal:
>> >>> >> >>
>> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
>> >>> >> >> len, int is_write)
>> >>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
>> >>> >> >>
>> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
>> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
>> >>> >> >>
>> >>> >> >>     I believe the qemu is trying to vist vram in this moment. This
>> >>> >> >> code seems no problem, and I will continue to find the root cause.
>> >>> >> >>
>> >>> >> >
>> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
>> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
>> >>> >> > xc_domain_populate_physmap_exact.
>> >>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
>> >>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
>> >>> >> > in your case?
>> >>> >>
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
>> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
>> >>> >>
>> >>> >> do_mmu_update
>> >>> >>       ->mod_l1_entry
>> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
>> >>> >>                          return -EINVAL;
>> >>> >>
>> >>> >>    mfn==INVALID_MFN, because :
>> >>> >>
>> >>> >> mod_l1_entry
>> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
>> >>> >>               ->p2m->get_entry
>> >>> >>                         ->p2m_gfn_to_mfn
>> >>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
>> >>> >>                                    /* This pfn is higher than the
>> >>> >> highest the p2m map currently holds */
>> >>> >>                                    return _mfn(INVALID_MFN);
>> >>> >>
>> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
>> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
>> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
>> >>> >> will exceed 0xfffff.
>> >>> >>
>> >>> >>    In the case of minios based stubdom, the mmu_update.vals do not
>> >>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
>> >>> >> exceeds 0x8000000100000000.
>> >>> >
>> >>> > It looks like the address of the guest that qemu is trying to map is not
>> >>> > valid.
>> >>> > Make sure you are running a guest with less than 2GB of ram, otherwise
>> >>> > you need the patch series that Anthony sent on Friday:
>> >>> >
>> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
>> >>>
>> >>> Not this problem. I never alloc more than 2GB for the hvm guest. The
>> >>> call stack in qemu is:
>> >>>
>> >>> qemu_get_ram_ptr
>> >>>       ->qemu_map_cache(addr, 0, 1)
>> >>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
>> >>> address_index ||
>> >>>                                           !test_bit(address_offset >>
>> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
>> >>>                            qemu_remap_bucket(entry, size ? :
>> >>> MCACHE_BUCKET_SIZE, address_index);
>> >>>                                  ->xc_map_foreign_bulk(xen_xc,
>> >>> xen_domid, PROT_READ|PROT_WRITE,
>> >>>
>> >>>                 pfns, err, nb_pfn);
>> >>>
>> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
>> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
>> >>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
>> >>> = 0xfffff)
>> >>>
>> >>>     if ( gfn > p2m->max_mapped_pfn )
>> >>>         /* This pfn is higher than the highest the p2m map currently holds */
>> >>>         return _mfn(INVALID_MFN);
>> >>>
>> >>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
>> >>> Maybe the address translation in linux stubdom cause this probem?
>> >>
>> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
>> >> does not exceed 2G:
>> >>
>> >> 0xfffff * 4096 > 2G
>> >>
>> >>
>> >>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
>> >>> needed in linux stubdom?
>> >>
>> >> hvmloader is the first thing that runs within the guest, it is not a
>> >> process in the stubdom or in dom0.
>> >> It is required in both minios and linux stubdoms.
>> >
>> > Hi Stefano,
>> >
>> >      I patched these patches, but we still have the same problem.
>> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
>> > vga_common_init function was also failed. Maybe this can explain the
>> > previous problem, which happened in the phase of trying to remap
>> > 0xc0000-0xc8fff of hvm guest into stubdom.
>> >
>> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
>> > located the failure in p2m_gfn_to_mfn function:
>> >
>> >     pod_retry_l3:
>> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
>> >        {
>> >                 .....
>> >                 return _mfn(INVALID_MFN);
>> >        }
>> >
>> >     I will continue to analyze this failure.
>> >
>> >     Thanks!
>> >
>> > Jiageng Yu.
>> >
>>
>>
>> Hi,
>>
>>     I compared the two executions of vga_common_init function in dom0
>> and linux based stubdom. The former succeeded and the later was
>> failed. First, they have the same call stack:
>>
>> Dom0 & Stubdom
>> _________________________________________________________
>> vga_common_init
>>      ->qemu_get_ram_ptr(s->vram_offset)
>>            ->block->host = xen_map_block(block->offset, block->length);
>>                  ->xc_map_foreign_bulk()
>>                         ->linux_privcmd_map_foreign_bulk()
>>                                ->xen_remap_domain_mfn_range()
>>                                      ->HYPERVISOR_mmu_update()
>> __________________________________________________________
>>
>> Xen
>> __________________________________________________________
>> do_mmu_update()
>>    ->case MMU_PT_UPDATE_PRESERVE_AD:
>>    ->case PGT_l1_page_table:
>>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v, pg_owner);
>>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
>> l1e_get_pfn(nl1e), &p2mt));
>>                  ->gfn_to_mfn_type_p2m()
>>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
>>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
>>                                       ->if ( (l3e_get_flags(*l3e) &
>> _PAGE_PRESENT) == 0 )
>>                                       ->    Error happens!
>>
>> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
>> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
>> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
>> address area(pfn:0x40000, size:8M) into linux stubdom's address space.
>>
>> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
>> gfn=0x40000, t=p2m_mmio_dm.
>> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
>> map_domain_page(mfn_x(mfn)) is also success. However, after executing:
>> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
>> the l3e->l3 =0 , and the error happens.
>>
>> So, in linux stubdom, when we are going to map the specified hvm
>> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
>> guest are not present. This is never happened in qemu of dom0. Could
>> you give me some prompts to this problem?
>
>
> It seems that you are trying to map pages that don't exist.
> The pages in question should be allocated by:
>
> qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
>    qemu_ram_alloc_from_ptr
>        xen_ram_alloc
>            xc_domain_populate_physmap_exact
>
> so I would add some printf and printk on this code path to find out if
> xc_domain_populate_physmap_exact fails for some reasons.

Hmm.. the linux stubdom kernel had a wrong p2m pair
<gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
xc_domain_populate_physmap_exact will setup the correct p2m pair
<gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
has not been updated, because the fllowing access to 0x40000 still
uses 0x127bd2.

I notice you have a patch: xen: modify kernel mappings corresponding
to granted pages. I think maybe it could slove my problem.

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 12325 bytes --]

On Wed, 27 Jul 2011, Jiageng Yu wrote:
> 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > On Tue, 26 Jul 2011, Jiageng Yu wrote:
> >> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> >> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
> >> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> >>> > CC'ing Tim and xen-devel
> >> >>> >
> >> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> >> >>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> >> >>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> >> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> >> >>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
> >> >>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
> >> >>> >> >> >>>
> >> >>> >> >> >>>
> >> >>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
> >> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
> >> >>> >> >> >>>
> >> >>> >> >> >>
> >> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
> >> >>> >> >> >> being loaded, it should be fixed now.
> >> >>> >> >> >>
> >> >>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
> >> >>> >> >> >>
> >> >>> >> >> >> ---
> >> >>> >> >> >>
> >> >>> >> >> >>
> >> >>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
> >> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
> >> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
> >> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> >> >>> >> >> >>             bios->create_pir_tables();
> >> >>> >> >> >>     }
> >> >>> >> >> >>
> >> >>> >> >> >> -    if ( bios->load_roms )
> >> >>> >> >> >> +    if ( 1 )
> >> >>> >> >> >>     {
> >> >>> >> >> >>         switch ( virtual_vga )
> >> >>> >> >> >>         {
> >> >>> >> >> >>
> >> >>> >> >> >>
> >> >>> >> >> >
> >> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
> >> >>> >> >> > signal subsequently. I am trying to print the call stack when
> >> >>> >> >> > receiving the signal.
> >> >>> >> >> >
> >> >>> >> >>
> >> >>> >> >> Hi,
> >> >>> >> >>
> >> >>> >> >>    I find the cause of SIGSEGV signal:
> >> >>> >> >>
> >> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
> >> >>> >> >> len, int is_write)
> >> >>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
> >> >>> >> >>
> >> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
> >> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
> >> >>> >> >>
> >> >>> >> >>     I believe the qemu is trying to vist vram in this moment. This
> >> >>> >> >> code seems no problem, and I will continue to find the root cause.
> >> >>> >> >>
> >> >>> >> >
> >> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
> >> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
> >> >>> >> > xc_domain_populate_physmap_exact.
> >> >>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
> >> >>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
> >> >>> >> > in your case?
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
> >> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> >> >>> >>
> >> >>> >> do_mmu_update
> >> >>> >>       ->mod_l1_entry
> >> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
> >> >>> >>                          return -EINVAL;
> >> >>> >>
> >> >>> >>    mfn==INVALID_MFN, because :
> >> >>> >>
> >> >>> >> mod_l1_entry
> >> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
> >> >>> >>               ->p2m->get_entry
> >> >>> >>                         ->p2m_gfn_to_mfn
> >> >>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
> >> >>> >>                                    /* This pfn is higher than the
> >> >>> >> highest the p2m map currently holds */
> >> >>> >>                                    return _mfn(INVALID_MFN);
> >> >>> >>
> >> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> >> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
> >> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
> >> >>> >> will exceed 0xfffff.
> >> >>> >>
> >> >>> >>    In the case of minios based stubdom, the mmu_update.vals do not
> >> >>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
> >> >>> >> exceeds 0x8000000100000000.
> >> >>> >
> >> >>> > It looks like the address of the guest that qemu is trying to map is not
> >> >>> > valid.
> >> >>> > Make sure you are running a guest with less than 2GB of ram, otherwise
> >> >>> > you need the patch series that Anthony sent on Friday:
> >> >>> >
> >> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> >> >>>
> >> >>> Not this problem. I never alloc more than 2GB for the hvm guest. The
> >> >>> call stack in qemu is:
> >> >>>
> >> >>> qemu_get_ram_ptr
> >> >>>       ->qemu_map_cache(addr, 0, 1)
> >> >>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
> >> >>> address_index ||
> >> >>>                                           !test_bit(address_offset >>
> >> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
> >> >>>                            qemu_remap_bucket(entry, size ? :
> >> >>> MCACHE_BUCKET_SIZE, address_index);
> >> >>>                                  ->xc_map_foreign_bulk(xen_xc,
> >> >>> xen_domid, PROT_READ|PROT_WRITE,
> >> >>>
> >> >>>                 pfns, err, nb_pfn);
> >> >>>
> >> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
> >> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
> >> >>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
> >> >>> = 0xfffff)
> >> >>>
> >> >>>     if ( gfn > p2m->max_mapped_pfn )
> >> >>>         /* This pfn is higher than the highest the p2m map currently holds */
> >> >>>         return _mfn(INVALID_MFN);
> >> >>>
> >> >>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
> >> >>> Maybe the address translation in linux stubdom cause this probem?
> >> >>
> >> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
> >> >> does not exceed 2G:
> >> >>
> >> >> 0xfffff * 4096 > 2G
> >> >>
> >> >>
> >> >>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
> >> >>> needed in linux stubdom?
> >> >>
> >> >> hvmloader is the first thing that runs within the guest, it is not a
> >> >> process in the stubdom or in dom0.
> >> >> It is required in both minios and linux stubdoms.
> >> >
> >> > Hi Stefano,
> >> >
> >> >      I patched these patches, but we still have the same problem.
> >> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> >> > vga_common_init function was also failed. Maybe this can explain the
> >> > previous problem, which happened in the phase of trying to remap
> >> > 0xc0000-0xc8fff of hvm guest into stubdom.
> >> >
> >> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
> >> > located the failure in p2m_gfn_to_mfn function:
> >> >
> >> >     pod_retry_l3:
> >> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
> >> >        {
> >> >                 .....
> >> >                 return _mfn(INVALID_MFN);
> >> >        }
> >> >
> >> >     I will continue to analyze this failure.
> >> >
> >> >     Thanks!
> >> >
> >> > Jiageng Yu.
> >> >
> >>
> >>
> >> Hi,
> >>
> >>     I compared the two executions of vga_common_init function in dom0
> >> and linux based stubdom. The former succeeded and the later was
> >> failed. First, they have the same call stack:
> >>
> >> Dom0 & Stubdom
> >> _________________________________________________________
> >> vga_common_init
> >>      ->qemu_get_ram_ptr(s->vram_offset)
> >>            ->block->host = xen_map_block(block->offset, block->length);
> >>                  ->xc_map_foreign_bulk()
> >>                         ->linux_privcmd_map_foreign_bulk()
> >>                                ->xen_remap_domain_mfn_range()
> >>                                      ->HYPERVISOR_mmu_update()
> >> __________________________________________________________
> >>
> >> Xen
> >> __________________________________________________________
> >> do_mmu_update()
> >>    ->case MMU_PT_UPDATE_PRESERVE_AD:
> >>    ->case PGT_l1_page_table:
> >>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v, pg_owner);
> >>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> >> l1e_get_pfn(nl1e), &p2mt));
> >>                  ->gfn_to_mfn_type_p2m()
> >>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
> >>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
> >>                                       ->if ( (l3e_get_flags(*l3e) &
> >> _PAGE_PRESENT) == 0 )
> >>                                       ->    Error happens!
> >>
> >> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
> >> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
> >> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
> >> address area(pfn:0x40000, size:8M) into linux stubdom's address space.
> >>
> >> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
> >> gfn=0x40000, t=p2m_mmio_dm.
> >> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
> >> map_domain_page(mfn_x(mfn)) is also success. However, after executing:
> >> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
> >> the l3e->l3 =0 , and the error happens.
> >>
> >> So, in linux stubdom, when we are going to map the specified hvm
> >> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
> >> guest are not present. This is never happened in qemu of dom0. Could
> >> you give me some prompts to this problem?
> >
> >
> > It seems that you are trying to map pages that don't exist.
> > The pages in question should be allocated by:
> >
> > qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
> >    qemu_ram_alloc_from_ptr
> >        xen_ram_alloc
> >            xc_domain_populate_physmap_exact
> >
> > so I would add some printf and printk on this code path to find out if
> > xc_domain_populate_physmap_exact fails for some reasons.
> 
> Hmm.. the linux stubdom kernel had a wrong p2m pair
> <gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
> xc_domain_populate_physmap_exact will setup the correct p2m pair
> <gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
> has not been updated, because the fllowing access to 0x40000 still
> uses 0x127bd2.

The p2m for the guest domain is only one in Xen, so I cannot understand
how it is possible that you get the old mfn value.
Also there shouldn't even be an old value because before
xc_domain_populate_physmap_exact pfn 0x40000 wasn't even allocated in
the guest yet.

Make sure you are using the right domid in both calls
(xc_domain_populate_physmap_exact and xc_map_foreign_bulk), also make
sure that libxenlight calls xc_domain_set_target and xs_set_target for
the stubdom otherwise the stubdom is not going to be privileged enough
to allocate and map memory of the guest.


> I notice you have a patch: xen: modify kernel mappings corresponding
> to granted pages. I think maybe it could slove my problem.

That patch fixes a different issue, related to grant table mappings.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1.1: Type: text/plain, Size: 14782 bytes --]

2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>

> On Wed, 27 Jul 2011, Jiageng Yu wrote:
> > 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > > On Tue, 26 Jul 2011, Jiageng Yu wrote:
> > >> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> > >> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
> > >> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >>> > CC'ing Tim and xen-devel
> > >> >>> >
> > >> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com
> >:
> > >> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> > >> >>> >> >> > 2011/7/15 Stefano Stabellini <
> stefano.stabellini@eu.citrix.com>:
> > >> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> >> >>> > Does it mean you are actually able to boot an HVM
> guest using Linux
> > >> >>> >> >> >>> > based stubdoms?? Did you manage to solve the
> framebuffer problem too?
> > >> >>> >> >> >>>
> > >> >>> >> >> >>>
> > >> >>> >> >> >>> The HVM guest is booted. But the boot process is
> terminated because
> > >> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck
> here for a week.
> > >> >>> >> >> >>>
> > >> >>> >> >> >>
> > >> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would
> prevent vga bios from
> > >> >>> >> >> >> being loaded, it should be fixed now.
> > >> >>> >> >> >>
> > >> >>> >> >> >> Alternatively you can temporarely work around the issue
> with this hacky patch:
> > >> >>> >> >> >>
> > >> >>> >> >> >> ---
> > >> >>> >> >> >>
> > >> >>> >> >> >>
> > >> >>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
> > >> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul
> 08 18:35:24 2011 +0100
> > >> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul
> 15 11:37:12 2011 +0000
> > >> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> > >> >>> >> >> >>             bios->create_pir_tables();
> > >> >>> >> >> >>     }
> > >> >>> >> >> >>
> > >> >>> >> >> >> -    if ( bios->load_roms )
> > >> >>> >> >> >> +    if ( 1 )
> > >> >>> >> >> >>     {
> > >> >>> >> >> >>         switch ( virtual_vga )
> > >> >>> >> >> >>         {
> > >> >>> >> >> >>
> > >> >>> >> >> >>
> > >> >>> >> >> >
> > >> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu
> receives a SIGSEGV
> > >> >>> >> >> > signal subsequently. I am trying to print the call stack
> when
> > >> >>> >> >> > receiving the signal.
> > >> >>> >> >> >
> > >> >>> >> >>
> > >> >>> >> >> Hi,
> > >> >>> >> >>
> > >> >>> >> >>    I find the cause of SIGSEGV signal:
> > >> >>> >> >>
> > >> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t
> *buf, int
> > >> >>> >> >> len, int is_write)
> > >> >>> >> >>                   ->memcpy(buf, ptr + (addr &
> ~TARGET_PAGE_MASK), l);
> > >> >>> >> >>
> > >> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts
> to vist
> > >> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
> > >> >>> >> >>
> > >> >>> >> >>     I believe the qemu is trying to vist vram in this
> moment. This
> > >> >>> >> >> code seems no problem, and I will continue to find the root
> cause.
> > >> >>> >> >>
> > >> >>> >> >
> > >> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
> > >> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that
> calls
> > >> >>> >> > xc_domain_populate_physmap_exact.
> > >> >>> >> > xc_domain_populate_physmap_exact is the hypercall that should
> ask Xen to
> > >> >>> >> > add the missing vram pages in the guest. Maybe this hypercall
> is failing
> > >> >>> >> > in your case?
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> Hi,
> > >> >>> >>
> > >> >>> >>    I continue to invesgate this bug and find
> hypercall_mmu_update in
> > >> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> > >> >>> >>
> > >> >>> >> do_mmu_update
> > >> >>> >>       ->mod_l1_entry
> > >> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn ==
> INVALID_MFN) )
> > >> >>> >>                          return -EINVAL;
> > >> >>> >>
> > >> >>> >>    mfn==INVALID_MFN, because :
> > >> >>> >>
> > >> >>> >> mod_l1_entry
> > >> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e),
> &p2mt));
> > >> >>> >>               ->p2m->get_entry
> > >> >>> >>                         ->p2m_gfn_to_mfn
> > >> >>> >>                                -> if ( gfn >
> p2m->max_mapped_pfn )
> > >> >>> >>                                    /* This pfn is higher than
> the
> > >> >>> >> highest the p2m map currently holds */
> > >> >>> >>                                    return _mfn(INVALID_MFN);
> > >> >>> >>
> > >> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> > >> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
> > >> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ).
> Therefore, gfn
> > >> >>> >> will exceed 0xfffff.
> > >> >>> >>
> > >> >>> >>    In the case of minios based stubdom, the mmu_update.vals do
> not
> > >> >>> >> exceed 0x8000000100000000. Next, I will invesgate why
> mmu_update.val
> > >> >>> >> exceeds 0x8000000100000000.
> > >> >>> >
> > >> >>> > It looks like the address of the guest that qemu is trying to
> map is not
> > >> >>> > valid.
> > >> >>> > Make sure you are running a guest with less than 2GB of ram,
> otherwise
> > >> >>> > you need the patch series that Anthony sent on Friday:
> > >> >>> >
> > >> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> > >> >>>
> > >> >>> Not this problem. I never alloc more than 2GB for the hvm guest.
> The
> > >> >>> call stack in qemu is:
> > >> >>>
> > >> >>> qemu_get_ram_ptr
> > >> >>>       ->qemu_map_cache(addr, 0, 1)
> > >> >>>                  -> if (!entry->vaddr_base || entry->paddr_index
> !=
> > >> >>> address_index ||
> > >> >>>                                           !test_bit(address_offset
> >>
> > >> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
> > >> >>>                            qemu_remap_bucket(entry, size ? :
> > >> >>> MCACHE_BUCKET_SIZE, address_index);
> > >> >>>                                  ->xc_map_foreign_bulk(xen_xc,
> > >> >>> xen_domid, PROT_READ|PROT_WRITE,
> > >> >>>
> > >> >>>                 pfns, err, nb_pfn);
> > >> >>>
> > >> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
> > >> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in
> the
> > >> >>> p2m_gfn_to_mfn, the judgement condition is
> valid:(p2m->max_mapped_pfn
> > >> >>> = 0xfffff)
> > >> >>>
> > >> >>>     if ( gfn > p2m->max_mapped_pfn )
> > >> >>>         /* This pfn is higher than the highest the p2m map
> currently holds */
> > >> >>>         return _mfn(INVALID_MFN);
> > >> >>>
> > >> >>>  In minios stubdom case, the hvm pages' pfns do not exceed
> 0xfffff.
> > >> >>> Maybe the address translation in linux stubdom cause this probem?
> > >> >>
> > >> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's
> memory
> > >> >> does not exceed 2G:
> > >> >>
> > >> >> 0xfffff * 4096 > 2G
> > >> >>
> > >> >>
> > >> >>>  BTW, in minios stubdom case, there seems no hvmloader process. Is
> it
> > >> >>> needed in linux stubdom?
> > >> >>
> > >> >> hvmloader is the first thing that runs within the guest, it is not
> a
> > >> >> process in the stubdom or in dom0.
> > >> >> It is required in both minios and linux stubdoms.
> > >> >
> > >> > Hi Stefano,
> > >> >
> > >> >      I patched these patches, but we still have the same problem.
> > >> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> > >> > vga_common_init function was also failed. Maybe this can explain the
> > >> > previous problem, which happened in the phase of trying to remap
> > >> > 0xc0000-0xc8fff of hvm guest into stubdom.
> > >> >
> > >> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset)
> and
> > >> > located the failure in p2m_gfn_to_mfn function:
> > >> >
> > >> >     pod_retry_l3:
> > >> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
> > >> >        {
> > >> >                 .....
> > >> >                 return _mfn(INVALID_MFN);
> > >> >        }
> > >> >
> > >> >     I will continue to analyze this failure.
> > >> >
> > >> >     Thanks!
> > >> >
> > >> > Jiageng Yu.
> > >> >
> > >>
> > >>
> > >> Hi,
> > >>
> > >>     I compared the two executions of vga_common_init function in dom0
> > >> and linux based stubdom. The former succeeded and the later was
> > >> failed. First, they have the same call stack:
> > >>
> > >> Dom0 & Stubdom
> > >> _________________________________________________________
> > >> vga_common_init
> > >>      ->qemu_get_ram_ptr(s->vram_offset)
> > >>            ->block->host = xen_map_block(block->offset,
> block->length);
> > >>                  ->xc_map_foreign_bulk()
> > >>                         ->linux_privcmd_map_foreign_bulk()
> > >>                                ->xen_remap_domain_mfn_range()
> > >>                                      ->HYPERVISOR_mmu_update()
> > >> __________________________________________________________
> > >>
> > >> Xen
> > >> __________________________________________________________
> > >> do_mmu_update()
> > >>    ->case MMU_PT_UPDATE_PRESERVE_AD:
> > >>    ->case PGT_l1_page_table:
> > >>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v,
> pg_owner);
> > >>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> > >> l1e_get_pfn(nl1e), &p2mt));
> > >>                  ->gfn_to_mfn_type_p2m()
> > >>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
> > >>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
> > >>                                       ->if ( (l3e_get_flags(*l3e) &
> > >> _PAGE_PRESENT) == 0 )
> > >>                                       ->    Error happens!
> > >>
> > >> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
> > >> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
> > >> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
> > >> address area(pfn:0x40000, size:8M) into linux stubdom's address space.
> > >>
> > >> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
> > >> gfn=0x40000, t=p2m_mmio_dm.
> > >> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
> > >> map_domain_page(mfn_x(mfn)) is also success. However, after executing:
> > >> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
> > >> the l3e->l3 =0 , and the error happens.
> > >>
> > >> So, in linux stubdom, when we are going to map the specified hvm
> > >> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
> > >> guest are not present. This is never happened in qemu of dom0. Could
> > >> you give me some prompts to this problem?
> > >
> > >
> > > It seems that you are trying to map pages that don't exist.
> > > The pages in question should be allocated by:
> > >
> > > qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
> > >    qemu_ram_alloc_from_ptr
> > >        xen_ram_alloc
> > >            xc_domain_populate_physmap_exact
> > >
> > > so I would add some printf and printk on this code path to find out if
> > > xc_domain_populate_physmap_exact fails for some reasons.
> >
> > Hmm.. the linux stubdom kernel had a wrong p2m pair
> > <gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
> > xc_domain_populate_physmap_exact will setup the correct p2m pair
> > <gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
> > has not been updated, because the fllowing access to 0x40000 still
> > uses 0x127bd2.
>
> The p2m for the guest domain is only one in Xen, so I cannot understand
> how it is possible that you get the old mfn value.
> Also there shouldn't even be an old value because before
> xc_domain_populate_physmap_exact pfn 0x40000 wasn't even allocated in
> the guest yet.
>
> Make sure you are using the right domid in both calls
> (xc_domain_populate_physmap_exact and xc_map_foreign_bulk), also make
> sure that libxenlight calls xc_domain_set_target and xs_set_target for
> the stubdom otherwise the stubdom is not going to be privileged enough
> to allocate and map memory of the guest.
>
>
> > I notice you have a patch: xen: modify kernel mappings corresponding
> > to granted pages. I think maybe it could slove my problem.
>
> That patch fixes a different issue, related to grant table mappings.



OK. That is my fault.

The root cause of previous problem is that the backend drivers in qemu are
not stopped. To confirm this root cause, I try to erase the codes about
stubdom in xen_be_init function of old qemu. The same problem appears. The
following patch is to fix this issue in upstream qemu.

diff --git a/xen-all.c b/xen-all.c
index b73fc43..8f0645e 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -472,12 +479,23 @@ static void cpu_handle_ioreq(void *opaque)
 static void xenstore_record_dm_state(XenIOState *s, const char *state)
 {
     char path[50];
+#ifdef CONFIG_STUBDOM
+    s->xenstore = xs_daemon_open();
+    if (s->xenstore == NULL) {
+        perror("xen: xenstore open");
+        return -errno;
+    }
+#endif
     snprintf(path, sizeof (path), "/local/domain/0/device-model/%u/state",
xen_domid);
     if (!xs_write(s->xenstore, XBT_NULL, path, state, strlen(state))) {
         fprintf(stderr, "error recording dm state\n");
         exit(1);
     }
+#ifdef CONFIG_STUBDOM
+    xs_daemon_close(s->xenstore);
+    s->xenstore = NULL;
+#endif
 }

 static void xen_main_loop_prepare(XenIOState *state)
@@ -538,6 +556,7 @@ int xen_hvm_init(void)

     state = qemu_mallocz(sizeof (XenIOState));

+#ifndef CONFIG_STUBDOM
     state->xce_handle = xen_xc_evtchn_open(NULL, 0);
     if (state->xce_handle == XC_HANDLER_INITIAL_VALUE) {
         perror("xen: event channel open");
@@ -549,6 +568,10 @@ int xen_hvm_init(void)
         perror("xen: xenstore open");
         return -errno;
     }
+#else
+    state->xce_handle = XC_HANDLER_INITIAL_VALUE;
+    state->xenstore = NULL;
+#endif

     state->exit.notify = xen_exit_notifier;
     qemu_add_exit_notifier(&state->exit);
@@ -575,9 +598,10 @@ int xen_hvm_init(void)

     state->ioreq_local_port = qemu_mallocz(smp_cpus * sizeof
(evtchn_port_t));

     /* FIXME: how about if we overflow the page here? */
     for (i = 0; i < smp_cpus; i++) {
-        rc = xc_evtchn_bind_interdomain(state->xce_handle, xen_domid,
+       rc = xc_evtchn_bind_interdomain(xen_xc, xen_domid,
                                         xen_vcpu_eport(state->shared_page,
i));
         if (rc == -1) {
             fprintf(stderr, "bind interdomain ioctl error %d\n", errno);

[-- Attachment #1.2: Type: text/html, Size: 22431 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1.1: Type: text/plain, Size: 16036 bytes --]

2011/7/28 Jiageng Yu <yujiageng734@gmail.com>

>
>
> 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>
>> On Wed, 27 Jul 2011, Jiageng Yu wrote:
>> > 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> > > On Tue, 26 Jul 2011, Jiageng Yu wrote:
>> > >> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
>> > >> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> > >> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
>> > >> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> > >> >>> > CC'ing Tim and xen-devel
>> > >> >>> >
>> > >> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
>> > >> >>> >> 2011/7/16 Stefano Stabellini <
>> stefano.stabellini@eu.citrix.com>:
>> > >> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> > >> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
>> > >> >>> >> >> > 2011/7/15 Stefano Stabellini <
>> stefano.stabellini@eu.citrix.com>:
>> > >> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
>> > >> >>> >> >> >>> > Does it mean you are actually able to boot an HVM
>> guest using Linux
>> > >> >>> >> >> >>> > based stubdoms?? Did you manage to solve the
>> framebuffer problem too?
>> > >> >>> >> >> >>>
>> > >> >>> >> >> >>>
>> > >> >>> >> >> >>> The HVM guest is booted. But the boot process is
>> terminated because
>> > >> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck
>> here for a week.
>> > >> >>> >> >> >>>
>> > >> >>> >> >> >>
>> > >> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would
>> prevent vga bios from
>> > >> >>> >> >> >> being loaded, it should be fixed now.
>> > >> >>> >> >> >>
>> > >> >>> >> >> >> Alternatively you can temporarely work around the issue
>> with this hacky patch:
>> > >> >>> >> >> >>
>> > >> >>> >> >> >> ---
>> > >> >>> >> >> >>
>> > >> >>> >> >> >>
>> > >> >>> >> >> >> diff -r 00d2c5ca26fd
>> tools/firmware/hvmloader/hvmloader.c
>> > >> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul
>> 08 18:35:24 2011 +0100
>> > >> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul
>> 15 11:37:12 2011 +0000
>> > >> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
>> > >> >>> >> >> >>             bios->create_pir_tables();
>> > >> >>> >> >> >>     }
>> > >> >>> >> >> >>
>> > >> >>> >> >> >> -    if ( bios->load_roms )
>> > >> >>> >> >> >> +    if ( 1 )
>> > >> >>> >> >> >>     {
>> > >> >>> >> >> >>         switch ( virtual_vga )
>> > >> >>> >> >> >>         {
>> > >> >>> >> >> >>
>> > >> >>> >> >> >>
>> > >> >>> >> >> >
>> > >> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu
>> receives a SIGSEGV
>> > >> >>> >> >> > signal subsequently. I am trying to print the call stack
>> when
>> > >> >>> >> >> > receiving the signal.
>> > >> >>> >> >> >
>> > >> >>> >> >>
>> > >> >>> >> >> Hi,
>> > >> >>> >> >>
>> > >> >>> >> >>    I find the cause of SIGSEGV signal:
>> > >> >>> >> >>
>> > >> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t
>> *buf, int
>> > >> >>> >> >> len, int is_write)
>> > >> >>> >> >>                   ->memcpy(buf, ptr + (addr &
>> ~TARGET_PAGE_MASK), l);
>> > >> >>> >> >>
>> > >> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts
>> to vist
>> > >> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
>> > >> >>> >> >>
>> > >> >>> >> >>     I believe the qemu is trying to vist vram in this
>> moment. This
>> > >> >>> >> >> code seems no problem, and I will continue to find the root
>> cause.
>> > >> >>> >> >>
>> > >> >>> >> >
>> > >> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
>> > >> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that
>> calls
>> > >> >>> >> > xc_domain_populate_physmap_exact.
>> > >> >>> >> > xc_domain_populate_physmap_exact is the hypercall that
>> should ask Xen to
>> > >> >>> >> > add the missing vram pages in the guest. Maybe this
>> hypercall is failing
>> > >> >>> >> > in your case?
>> > >> >>> >>
>> > >> >>> >>
>> > >> >>> >> Hi,
>> > >> >>> >>
>> > >> >>> >>    I continue to invesgate this bug and find
>> hypercall_mmu_update in
>> > >> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
>> > >> >>> >>
>> > >> >>> >> do_mmu_update
>> > >> >>> >>       ->mod_l1_entry
>> > >> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn ==
>> INVALID_MFN) )
>> > >> >>> >>                          return -EINVAL;
>> > >> >>> >>
>> > >> >>> >>    mfn==INVALID_MFN, because :
>> > >> >>> >>
>> > >> >>> >> mod_l1_entry
>> > >> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e),
>> &p2mt));
>> > >> >>> >>               ->p2m->get_entry
>> > >> >>> >>                         ->p2m_gfn_to_mfn
>> > >> >>> >>                                -> if ( gfn >
>> p2m->max_mapped_pfn )
>> > >> >>> >>                                    /* This pfn is higher than
>> the
>> > >> >>> >> highest the p2m map currently holds */
>> > >> >>> >>                                    return _mfn(INVALID_MFN);
>> > >> >>> >>
>> > >> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
>> > >> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e
>> =
>> > >> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ).
>> Therefore, gfn
>> > >> >>> >> will exceed 0xfffff.
>> > >> >>> >>
>> > >> >>> >>    In the case of minios based stubdom, the mmu_update.vals do
>> not
>> > >> >>> >> exceed 0x8000000100000000. Next, I will invesgate why
>> mmu_update.val
>> > >> >>> >> exceeds 0x8000000100000000.
>> > >> >>> >
>> > >> >>> > It looks like the address of the guest that qemu is trying to
>> map is not
>> > >> >>> > valid.
>> > >> >>> > Make sure you are running a guest with less than 2GB of ram,
>> otherwise
>> > >> >>> > you need the patch series that Anthony sent on Friday:
>> > >> >>> >
>> > >> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
>> > >> >>>
>> > >> >>> Not this problem. I never alloc more than 2GB for the hvm guest.
>> The
>> > >> >>> call stack in qemu is:
>> > >> >>>
>> > >> >>> qemu_get_ram_ptr
>> > >> >>>       ->qemu_map_cache(addr, 0, 1)
>> > >> >>>                  -> if (!entry->vaddr_base || entry->paddr_index
>> !=
>> > >> >>> address_index ||
>> > >> >>>
>> !test_bit(address_offset >>
>> > >> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
>> > >> >>>                            qemu_remap_bucket(entry, size ? :
>> > >> >>> MCACHE_BUCKET_SIZE, address_index);
>> > >> >>>                                  ->xc_map_foreign_bulk(xen_xc,
>> > >> >>> xen_domid, PROT_READ|PROT_WRITE,
>> > >> >>>
>> > >> >>>                 pfns, err, nb_pfn);
>> > >> >>>
>> > >> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
>> > >> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in
>> the
>> > >> >>> p2m_gfn_to_mfn, the judgement condition is
>> valid:(p2m->max_mapped_pfn
>> > >> >>> = 0xfffff)
>> > >> >>>
>> > >> >>>     if ( gfn > p2m->max_mapped_pfn )
>> > >> >>>         /* This pfn is higher than the highest the p2m map
>> currently holds */
>> > >> >>>         return _mfn(INVALID_MFN);
>> > >> >>>
>> > >> >>>  In minios stubdom case, the hvm pages' pfns do not exceed
>> 0xfffff.
>> > >> >>> Maybe the address translation in linux stubdom cause this probem?
>> > >> >>
>> > >> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's
>> memory
>> > >> >> does not exceed 2G:
>> > >> >>
>> > >> >> 0xfffff * 4096 > 2G
>> > >> >>
>> > >> >>
>> > >> >>>  BTW, in minios stubdom case, there seems no hvmloader process.
>> Is it
>> > >> >>> needed in linux stubdom?
>> > >> >>
>> > >> >> hvmloader is the first thing that runs within the guest, it is not
>> a
>> > >> >> process in the stubdom or in dom0.
>> > >> >> It is required in both minios and linux stubdoms.
>> > >> >
>> > >> > Hi Stefano,
>> > >> >
>> > >> >      I patched these patches, but we still have the same problem.
>> > >> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
>> > >> > vga_common_init function was also failed. Maybe this can explain
>> the
>> > >> > previous problem, which happened in the phase of trying to remap
>> > >> > 0xc0000-0xc8fff of hvm guest into stubdom.
>> > >> >
>> > >> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset)
>> and
>> > >> > located the failure in p2m_gfn_to_mfn function:
>> > >> >
>> > >> >     pod_retry_l3:
>> > >> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
>> > >> >        {
>> > >> >                 .....
>> > >> >                 return _mfn(INVALID_MFN);
>> > >> >        }
>> > >> >
>> > >> >     I will continue to analyze this failure.
>> > >> >
>> > >> >     Thanks!
>> > >> >
>> > >> > Jiageng Yu.
>> > >> >
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >>     I compared the two executions of vga_common_init function in dom0
>> > >> and linux based stubdom. The former succeeded and the later was
>> > >> failed. First, they have the same call stack:
>> > >>
>> > >> Dom0 & Stubdom
>> > >> _________________________________________________________
>> > >> vga_common_init
>> > >>      ->qemu_get_ram_ptr(s->vram_offset)
>> > >>            ->block->host = xen_map_block(block->offset,
>> block->length);
>> > >>                  ->xc_map_foreign_bulk()
>> > >>                         ->linux_privcmd_map_foreign_bulk()
>> > >>                                ->xen_remap_domain_mfn_range()
>> > >>                                      ->HYPERVISOR_mmu_update()
>> > >> __________________________________________________________
>> > >>
>> > >> Xen
>> > >> __________________________________________________________
>> > >> do_mmu_update()
>> > >>    ->case MMU_PT_UPDATE_PRESERVE_AD:
>> > >>    ->case PGT_l1_page_table:
>> > >>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v,
>> pg_owner);
>> > >>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
>> > >> l1e_get_pfn(nl1e), &p2mt));
>> > >>                  ->gfn_to_mfn_type_p2m()
>> > >>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
>> > >>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
>> > >>                                       ->if ( (l3e_get_flags(*l3e) &
>> > >> _PAGE_PRESENT) == 0 )
>> > >>                                       ->    Error happens!
>> > >>
>> > >> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
>> > >> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
>> > >> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
>> > >> address area(pfn:0x40000, size:8M) into linux stubdom's address
>> space.
>> > >>
>> > >> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
>> > >> gfn=0x40000, t=p2m_mmio_dm.
>> > >> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
>> > >> map_domain_page(mfn_x(mfn)) is also success. However, after
>> executing:
>> > >> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
>> > >> the l3e->l3 =0 , and the error happens.
>> > >>
>> > >> So, in linux stubdom, when we are going to map the specified hvm
>> > >> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
>> > >> guest are not present. This is never happened in qemu of dom0. Could
>> > >> you give me some prompts to this problem?
>> > >
>> > >
>> > > It seems that you are trying to map pages that don't exist.
>> > > The pages in question should be allocated by:
>> > >
>> > > qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
>> > >    qemu_ram_alloc_from_ptr
>> > >        xen_ram_alloc
>> > >            xc_domain_populate_physmap_exact
>> > >
>> > > so I would add some printf and printk on this code path to find out if
>> > > xc_domain_populate_physmap_exact fails for some reasons.
>> >
>> > Hmm.. the linux stubdom kernel had a wrong p2m pair
>> > <gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
>> > xc_domain_populate_physmap_exact will setup the correct p2m pair
>> > <gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
>> > has not been updated, because the fllowing access to 0x40000 still
>> > uses 0x127bd2.
>>
>> The p2m for the guest domain is only one in Xen, so I cannot understand
>> how it is possible that you get the old mfn value.
>> Also there shouldn't even be an old value because before
>> xc_domain_populate_physmap_exact pfn 0x40000 wasn't even allocated in
>> the guest yet.
>>
>> Make sure you are using the right domid in both calls
>> (xc_domain_populate_physmap_exact and xc_map_foreign_bulk), also make
>> sure that libxenlight calls xc_domain_set_target and xs_set_target for
>> the stubdom otherwise the stubdom is not going to be privileged enough
>> to allocate and map memory of the guest.
>>
>>
>> > I notice you have a patch: xen: modify kernel mappings corresponding
>> > to granted pages. I think maybe it could slove my problem.
>>
>> That patch fixes a different issue, related to grant table mappings.
>
>
>
> OK. That is my fault.
>
> The root cause of previous problem is that the backend drivers in qemu are
> not stopped. To confirm this root cause, I try to erase the codes about
> stubdom in xen_be_init function of old qemu. The same problem appears. The
> following patch is to fix this issue in upstream qemu.
>
> diff --git a/xen-all.c b/xen-all.c
> index b73fc43..8f0645e 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -472,12 +479,23 @@ static void cpu_handle_ioreq(void *opaque)
>  static void xenstore_record_dm_state(XenIOState *s, const char *state)
>  {
>      char path[50];
> +#ifdef CONFIG_STUBDOM
> +    s->xenstore = xs_daemon_open();
> +    if (s->xenstore == NULL) {
> +        perror("xen: xenstore open");
> +        return -errno;
> +    }
> +#endif
>      snprintf(path, sizeof (path), "/local/domain/0/device-model/%u/state",
> xen_domid);
>      if (!xs_write(s->xenstore, XBT_NULL, path, state, strlen(state))) {
>          fprintf(stderr, "error recording dm state\n");
>          exit(1);
>      }
> +#ifdef CONFIG_STUBDOM
> +    xs_daemon_close(s->xenstore);
> +    s->xenstore = NULL;
> +#endif
>  }
>
>  static void xen_main_loop_prepare(XenIOState *state)
> @@ -538,6 +556,7 @@ int xen_hvm_init(void)
>
>      state = qemu_mallocz(sizeof (XenIOState));
>
> +#ifndef CONFIG_STUBDOM
>      state->xce_handle = xen_xc_evtchn_open(NULL, 0);
>      if (state->xce_handle == XC_HANDLER_INITIAL_VALUE) {
>          perror("xen: event channel open");
> @@ -549,6 +568,10 @@ int xen_hvm_init(void)
>          perror("xen: xenstore open");
>          return -errno;
>      }
> +#else
> +    state->xce_handle = XC_HANDLER_INITIAL_VALUE;
> +    state->xenstore = NULL;
> +#endif
>
>      state->exit.notify = xen_exit_notifier;
>      qemu_add_exit_notifier(&state->exit);
> @@ -575,9 +598,10 @@ int xen_hvm_init(void)
>
>      state->ioreq_local_port = qemu_mallocz(smp_cpus * sizeof
> (evtchn_port_t));
>
>      /* FIXME: how about if we overflow the page here? */
>      for (i = 0; i < smp_cpus; i++) {
> -        rc = xc_evtchn_bind_interdomain(state->xce_handle, xen_domid,
> +       rc = xc_evtchn_bind_interdomain(xen_xc, xen_domid,
>                                          xen_vcpu_eport(state->shared_page,
> i));
>          if (rc == -1) {
>              fprintf(stderr, "bind interdomain ioctl error %d\n", errno);
>


The new problem is my stubdom hangs at:

hvmloader:
     ->main()
            ->pci_setup()
                    ->pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);

I am investigating this problem. The pci_writeb will finally call the
hvm_set_pci_link_route in Xen:

hvmloader:             pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);
qemu(stubdom):     PCIHostState->data_handler->write()
qemu(stubdom):     i440fx_write_config_xen()
qemu(stubdom):     xen_piix_pci_write_config_client()
xenctrl:                   xc_hvm_set_pci_link_route()

The ioport is registered by pci_host_data_register_ioport(0xcfc, s)
function.

I will find out why not invoke i440fx_write_config_xen() in my case. I will
also read the pciutils.patch of minios stubdom and maybe find something
interesting.

[-- Attachment #1.2: Type: text/html, Size: 24004 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 17203 bytes --]

On Thu, 28 Jul 2011, Jiageng Yu wrote:
> 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> On Wed, 27 Jul 2011, Jiageng Yu wrote:
> > 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > > On Tue, 26 Jul 2011, Jiageng Yu wrote:
> > >> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> > >> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
> > >> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >>> > CC'ing Tim and xen-devel
> > >> >>> >
> > >> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> 2011/7/16 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> > >> >>> >> >> > 2011/7/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > >> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > >> >>> >> >> >>> > Does it mean you are actually able to boot an HVM guest using Linux
> > >> >>> >> >> >>> > based stubdoms?? Did you manage to solve the framebuffer problem too?
> > >> >>> >> >> >>>
> > >> >>> >> >> >>>
> > >> >>> >> >> >>> The HVM guest is booted. But the boot process is terminated because
> > >> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck here for a week.
> > >> >>> >> >> >>>
> > >> >>> >> >> >>
> > >> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that would prevent vga bios from
> > >> >>> >> >> >> being loaded, it should be fixed now.
> > >> >>> >> >> >>
> > >> >>> >> >> >> Alternatively you can temporarely work around the issue with this hacky patch:
> > >> >>> >> >> >>
> > >> >>> >> >> >> ---
> > >> >>> >> >> >>
> > >> >>> >> >> >>
> > >> >>> >> >> >> diff -r 00d2c5ca26fd tools/firmware/hvmloader/hvmloader.c
> > >> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul 08 18:35:24 2011 +0100
> > >> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul 15 11:37:12 2011 +0000
> > >> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> > >> >>> >> >> >>             bios->create_pir_tables();
> > >> >>> >> >> >>     }
> > >> >>> >> >> >>
> > >> >>> >> >> >> -    if ( bios->load_roms )
> > >> >>> >> >> >> +    if ( 1 )
> > >> >>> >> >> >>     {
> > >> >>> >> >> >>         switch ( virtual_vga )
> > >> >>> >> >> >>         {
> > >> >>> >> >> >>
> > >> >>> >> >> >>
> > >> >>> >> >> >
> > >> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu receives a SIGSEGV
> > >> >>> >> >> > signal subsequently. I am trying to print the call stack when
> > >> >>> >> >> > receiving the signal.
> > >> >>> >> >> >
> > >> >>> >> >>
> > >> >>> >> >> Hi,
> > >> >>> >> >>
> > >> >>> >> >>    I find the cause of SIGSEGV signal:
> > >> >>> >> >>
> > >> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t *buf, int
> > >> >>> >> >> len, int is_write)
> > >> >>> >> >>                   ->memcpy(buf, ptr + (addr & ~TARGET_PAGE_MASK), l);
> > >> >>> >> >>
> > >> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts to vist
> > >> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
> > >> >>> >> >>
> > >> >>> >> >>     I believe the qemu is trying to vist vram in this moment. This
> > >> >>> >> >> code seems no problem, and I will continue to find the root cause.
> > >> >>> >> >>
> > >> >>> >> >
> > >> >>> >> > The vram is allocated by qemu, see hw/vga.c:vga_common_init.
> > >> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that calls
> > >> >>> >> > xc_domain_populate_physmap_exact.
> > >> >>> >> > xc_domain_populate_physmap_exact is the hypercall that should ask Xen to
> > >> >>> >> > add the missing vram pages in the guest. Maybe this hypercall is failing
> > >> >>> >> > in your case?
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> Hi,
> > >> >>> >>
> > >> >>> >>    I continue to invesgate this bug and find hypercall_mmu_update in
> > >> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> > >> >>> >>
> > >> >>> >> do_mmu_update
> > >> >>> >>       ->mod_l1_entry
> > >> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn == INVALID_MFN) )
> > >> >>> >>                          return -EINVAL;
> > >> >>> >>
> > >> >>> >>    mfn==INVALID_MFN, because :
> > >> >>> >>
> > >> >>> >> mod_l1_entry
> > >> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom), l1e_get_pfn(nl1e), &p2mt));
> > >> >>> >>               ->p2m->get_entry
> > >> >>> >>                         ->p2m_gfn_to_mfn
> > >> >>> >>                                -> if ( gfn > p2m->max_mapped_pfn )
> > >> >>> >>                                    /* This pfn is higher than the
> > >> >>> >> highest the p2m map currently holds */
> > >> >>> >>                                    return _mfn(INVALID_MFN);
> > >> >>> >>
> > >> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> > >> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e =
> > >> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ). Therefore, gfn
> > >> >>> >> will exceed 0xfffff.
> > >> >>> >>
> > >> >>> >>    In the case of minios based stubdom, the mmu_update.vals do not
> > >> >>> >> exceed 0x8000000100000000. Next, I will invesgate why mmu_update.val
> > >> >>> >> exceeds 0x8000000100000000.
> > >> >>> >
> > >> >>> > It looks like the address of the guest that qemu is trying to map is not
> > >> >>> > valid.
> > >> >>> > Make sure you are running a guest with less than 2GB of ram, otherwise
> > >> >>> > you need the patch series that Anthony sent on Friday:
> > >> >>> >
> > >> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> > >> >>>
> > >> >>> Not this problem. I never alloc more than 2GB for the hvm guest. The
> > >> >>> call stack in qemu is:
> > >> >>>
> > >> >>> qemu_get_ram_ptr
> > >> >>>       ->qemu_map_cache(addr, 0, 1)
> > >> >>>                  -> if (!entry->vaddr_base || entry->paddr_index !=
> > >> >>> address_index ||
> > >> >>>                                           !test_bit(address_offset >>
> > >> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
> > >> >>>                            qemu_remap_bucket(entry, size ? :
> > >> >>> MCACHE_BUCKET_SIZE, address_index);
> > >> >>>                                  ->xc_map_foreign_bulk(xen_xc,
> > >> >>> xen_domid, PROT_READ|PROT_WRITE,
> > >> >>>
> > >> >>>                 pfns, err, nb_pfn);
> > >> >>>
> > >> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
> > >> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So, in the
> > >> >>> p2m_gfn_to_mfn, the judgement condition is valid:(p2m->max_mapped_pfn
> > >> >>> = 0xfffff)
> > >> >>>
> > >> >>>     if ( gfn > p2m->max_mapped_pfn )
> > >> >>>         /* This pfn is higher than the highest the p2m map currently holds */
> > >> >>>         return _mfn(INVALID_MFN);
> > >> >>>
> > >> >>>  In minios stubdom case, the hvm pages' pfns do not exceed 0xfffff.
> > >> >>> Maybe the address translation in linux stubdom cause this probem?
> > >> >>
> > >> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's memory
> > >> >> does not exceed 2G:
> > >> >>
> > >> >> 0xfffff * 4096 > 2G
> > >> >>
> > >> >>
> > >> >>>  BTW, in minios stubdom case, there seems no hvmloader process. Is it
> > >> >>> needed in linux stubdom?
> > >> >>
> > >> >> hvmloader is the first thing that runs within the guest, it is not a
> > >> >> process in the stubdom or in dom0.
> > >> >> It is required in both minios and linux stubdoms.
> > >> >
> > >> > Hi Stefano,
> > >> >
> > >> >      I patched these patches, but we still have the same problem.
> > >> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> > >> > vga_common_init function was also failed. Maybe this can explain the
> > >> > previous problem, which happened in the phase of trying to remap
> > >> > 0xc0000-0xc8fff of hvm guest into stubdom.
> > >> >
> > >> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset) and
> > >> > located the failure in p2m_gfn_to_mfn function:
> > >> >
> > >> >     pod_retry_l3:
> > >> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
> > >> >        {
> > >> >                 .....
> > >> >                 return _mfn(INVALID_MFN);
> > >> >        }
> > >> >
> > >> >     I will continue to analyze this failure.
> > >> >
> > >> >     Thanks!
> > >> >
> > >> > Jiageng Yu.
> > >> >
> > >>
> > >>
> > >> Hi,
> > >>
> > >>     I compared the two executions of vga_common_init function in dom0
> > >> and linux based stubdom. The former succeeded and the later was
> > >> failed. First, they have the same call stack:
> > >>
> > >> Dom0 & Stubdom
> > >> _________________________________________________________
> > >> vga_common_init
> > >>      ->qemu_get_ram_ptr(s->vram_offset)
> > >>            ->block->host = xen_map_block(block->offset, block->length);
> > >>                  ->xc_map_foreign_bulk()
> > >>                         ->linux_privcmd_map_foreign_bulk()
> > >>                                ->xen_remap_domain_mfn_range()
> > >>                                      ->HYPERVISOR_mmu_update()
> > >> __________________________________________________________
> > >>
> > >> Xen
> > >> __________________________________________________________
> > >> do_mmu_update()
> > >>    ->case MMU_PT_UPDATE_PRESERVE_AD:
> > >>    ->case PGT_l1_page_table:
> > >>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v, pg_owner);
> > >>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> > >> l1e_get_pfn(nl1e), &p2mt));
> > >>                  ->gfn_to_mfn_type_p2m()
> > >>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
> > >>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
> > >>                                       ->if ( (l3e_get_flags(*l3e) &
> > >> _PAGE_PRESENT) == 0 )
> > >>                                       ->    Error happens!
> > >>
> > >> The qemu in dom0 can find the l3e of hvm guest, but the qemu in linux
> > >> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
> > >> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
> > >> address area(pfn:0x40000, size:8M) into linux stubdom's address space.
> > >>
> > >> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
> > >> gfn=0x40000, t=p2m_mmio_dm.
> > >> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
> > >> map_domain_page(mfn_x(mfn)) is also success. However, after executing:
> > >> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
> > >> the l3e->l3 =0 , and the error happens.
> > >>
> > >> So, in linux stubdom, when we are going to map the specified hvm
> > >> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
> > >> guest are not present. This is never happened in qemu of dom0. Could
> > >> you give me some prompts to this problem?
> > >
> > >
> > > It seems that you are trying to map pages that don't exist.
> > > The pages in question should be allocated by:
> > >
> > > qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
> > >    qemu_ram_alloc_from_ptr
> > >        xen_ram_alloc
> > >            xc_domain_populate_physmap_exact
> > >
> > > so I would add some printf and printk on this code path to find out if
> > > xc_domain_populate_physmap_exact fails for some reasons.
> >
> > Hmm.. the linux stubdom kernel had a wrong p2m pair
> > <gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
> > xc_domain_populate_physmap_exact will setup the correct p2m pair
> > <gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
> > has not been updated, because the fllowing access to 0x40000 still
> > uses 0x127bd2.
> 
> The p2m for the guest domain is only one in Xen, so I cannot understand
> how it is possible that you get the old mfn value.
> Also there shouldn't even be an old value because before
> xc_domain_populate_physmap_exact pfn 0x40000 wasn't even allocated in
> the guest yet.
> 
> Make sure you are using the right domid in both calls
> (xc_domain_populate_physmap_exact and xc_map_foreign_bulk), also make
> sure that libxenlight calls xc_domain_set_target and xs_set_target for
> the stubdom otherwise the stubdom is not going to be privileged enough
> to allocate and map memory of the guest.
> 
> 
> > I notice you have a patch: xen: modify kernel mappings corresponding
> > to granted pages. I think maybe it could slove my problem.
> 
> That patch fixes a different issue, related to grant table mappings.
> 
> 
>  
> OK. That is my fault.
>  
> The root cause of previous problem is that the backend drivers in qemu are not stopped. To confirm this root cause,
> I try to erase the codes about stubdom in xen_be_init function of old qemu. The same problem appears. The following
> patch is to fix this issue in upstream qemu.
>  
> diff --git a/xen-all.c b/xen-all.c
> index b73fc43..8f0645e 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -472,12 +479,23 @@ static void cpu_handle_ioreq(void *opaque)
>  static void xenstore_record_dm_state(XenIOState *s, const char *state)
>  {
>      char path[50];
> +#ifdef CONFIG_STUBDOM
> +    s->xenstore = xs_daemon_open();
> +    if (s->xenstore == NULL) {
> +        perror("xen: xenstore open");
> +        return -errno;
> +    }
> +#endif
>      snprintf(path, sizeof (path), "/local/domain/0/device-model/%u/state", xen_domid);
>      if (!xs_write(s->xenstore, XBT_NULL, path, state, strlen(state))) {
>          fprintf(stderr, "error recording dm state\n");
>          exit(1);
>      }
> +#ifdef CONFIG_STUBDOM
> +    xs_daemon_close(s->xenstore);
> +    s->xenstore = NULL;
> +#endif
>  }

Why do you need to re-open the xenstore connection here?
It should be already been opened by xen_hvm_init, like in the normal
case.


>  static void xen_main_loop_prepare(XenIOState *state)
> @@ -538,6 +556,7 @@ int xen_hvm_init(void)
>  
>      state = qemu_mallocz(sizeof (XenIOState));
>  
> +#ifndef CONFIG_STUBDOM
>      state->xce_handle = xen_xc_evtchn_open(NULL, 0);
>      if (state->xce_handle == XC_HANDLER_INITIAL_VALUE) {
>          perror("xen: event channel open");
> @@ -549,6 +568,10 @@ int xen_hvm_init(void)
>          perror("xen: xenstore open");
>          return -errno;
>      }
> +#else
> +    state->xce_handle = XC_HANDLER_INITIAL_VALUE;
> +    state->xenstore = NULL;
> +#endif
>  

So you are explicitly avoiding to open the xenstore connection from
xen_hvm_init, why?
I think you might be trying to fix a race condition, maybe something is
not ready yet at this point that becomes ready later?


>      state->exit.notify = xen_exit_notifier;
>      qemu_add_exit_notifier(&state->exit);
> @@ -575,9 +598,10 @@ int xen_hvm_init(void)
>  
>      state->ioreq_local_port = qemu_mallocz(smp_cpus * sizeof (evtchn_port_t));
>  
>      /* FIXME: how about if we overflow the page here? */
>      for (i = 0; i < smp_cpus; i++) {
> -        rc = xc_evtchn_bind_interdomain(state->xce_handle, xen_domid,
> +       rc = xc_evtchn_bind_interdomain(xen_xc, xen_domid,
>                                          xen_vcpu_eport(state->shared_page, i));
>          if (rc == -1) {
>              fprintf(stderr, "bind interdomain ioctl error %d\n", errno);
> 

This cannot be right: xc_evtchn_bind_interdomain takes a xc_evtchn* as
first paramter while xen_xc is xc_interface*
This change would prevent you from receiving any IO request
notifications from Xen.

 
> The new problem is my stubdom hangs at:
>  
> hvmloader:
>      ->main()
>             ->pci_setup()
>                     ->pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);
>  
> I am investigating this problem. The pci_writeb will finally call the hvm_set_pci_link_route in Xen:
>  
> hvmloader:             pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);
> qemu(stubdom):     PCIHostState->data_handler->write()
> qemu(stubdom):     i440fx_write_config_xen()
> qemu(stubdom):     xen_piix_pci_write_config_client()
> xenctrl:                   xc_hvm_set_pci_link_route()
>  
> The ioport is registered by pci_host_data_register_ioport(0xcfc, s) function.
>  
> I will find out why not invoke i440fx_write_config_xen() in my case. I will also read the pciutils.patch of minios stubdom
> and maybe find something interesting.
 
I think you are not receiving any IO request notifications from Xen
because of the previous change.
It is probably worth adding a printf into xen-all.c:handle_ioreq to see
if you receive something.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1.1: Type: text/plain, Size: 21253 bytes --]

2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>

> On Thu, 28 Jul 2011, Jiageng Yu wrote:
> > 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > On Wed, 27 Jul 2011, Jiageng Yu wrote:
> > > 2011/7/27 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > > > On Tue, 26 Jul 2011, Jiageng Yu wrote:
> > > >> 2011/7/26 Jiageng Yu <yujiageng734@gmail.com>:
> > > >> > 2011/7/22 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> > > >> >> On Thu, 21 Jul 2011, Jiageng Yu wrote:
> > > >> >>> 2011/7/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com
> >:
> > > >> >>> > CC'ing Tim and xen-devel
> > > >> >>> >
> > > >> >>> > On Mon, 18 Jul 2011, Jiageng Yu wrote:
> > > >> >>> >> 2011/7/16 Stefano Stabellini <
> stefano.stabellini@eu.citrix.com>:
> > > >> >>> >> > On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > > >> >>> >> >> 2011/7/15 Jiageng Yu <yujiageng734@gmail.com>:
> > > >> >>> >> >> > 2011/7/15 Stefano Stabellini <
> stefano.stabellini@eu.citrix.com>:
> > > >> >>> >> >> >> On Fri, 15 Jul 2011, Jiageng Yu wrote:
> > > >> >>> >> >> >>> > Does it mean you are actually able to boot an HVM
> guest using Linux
> > > >> >>> >> >> >>> > based stubdoms?? Did you manage to solve the
> framebuffer problem too?
> > > >> >>> >> >> >>>
> > > >> >>> >> >> >>>
> > > >> >>> >> >> >>> The HVM guest is booted. But the boot process is
> terminated because
> > > >> >>> >> >> >>> vga bios is not invoked by seabios. I have got stuck
> here for a week.
> > > >> >>> >> >> >>>
> > > >> >>> >> >> >>
> > > >> >>> >> >> >> There was a bug in xen-unstable.hg or seabios that
> would prevent vga bios from
> > > >> >>> >> >> >> being loaded, it should be fixed now.
> > > >> >>> >> >> >>
> > > >> >>> >> >> >> Alternatively you can temporarely work around the issue
> with this hacky patch:
> > > >> >>> >> >> >>
> > > >> >>> >> >> >> ---
> > > >> >>> >> >> >>
> > > >> >>> >> >> >>
> > > >> >>> >> >> >> diff -r 00d2c5ca26fd
> tools/firmware/hvmloader/hvmloader.c
> > > >> >>> >> >> >> --- a/tools/firmware/hvmloader/hvmloader.c      Fri Jul
> 08 18:35:24 2011 +0100
> > > >> >>> >> >> >> +++ b/tools/firmware/hvmloader/hvmloader.c      Fri Jul
> 15 11:37:12 2011 +0000
> > > >> >>> >> >> >> @@ -430,7 +430,7 @@ int main(void)
> > > >> >>> >> >> >>             bios->create_pir_tables();
> > > >> >>> >> >> >>     }
> > > >> >>> >> >> >>
> > > >> >>> >> >> >> -    if ( bios->load_roms )
> > > >> >>> >> >> >> +    if ( 1 )
> > > >> >>> >> >> >>     {
> > > >> >>> >> >> >>         switch ( virtual_vga )
> > > >> >>> >> >> >>         {
> > > >> >>> >> >> >>
> > > >> >>> >> >> >>
> > > >> >>> >> >> >
> > > >> >>> >> >> > Yes. Vga bios is booted. However, the upstram qemu
> receives a SIGSEGV
> > > >> >>> >> >> > signal subsequently. I am trying to print the call stack
> when
> > > >> >>> >> >> > receiving the signal.
> > > >> >>> >> >> >
> > > >> >>> >> >>
> > > >> >>> >> >> Hi,
> > > >> >>> >> >>
> > > >> >>> >> >>    I find the cause of SIGSEGV signal:
> > > >> >>> >> >>
> > > >> >>> >> >>    cpu_physical_memory_rw(target_phys_addr_t addr, uint8_t
> *buf, int
> > > >> >>> >> >> len, int is_write)
> > > >> >>> >> >>                   ->memcpy(buf, ptr + (addr &
> ~TARGET_PAGE_MASK), l);
> > > >> >>> >> >>
> > > >> >>> >> >>     In my case, ptr=0 and addr=0xc253e, when qemu attempts
> to vist
> > > >> >>> >> >> 0x53e address, the SIGSEGV signal is generated.
> > > >> >>> >> >>
> > > >> >>> >> >>     I believe the qemu is trying to vist vram in this
> moment. This
> > > >> >>> >> >> code seems no problem, and I will continue to find the
> root cause.
> > > >> >>> >> >>
> > > >> >>> >> >
> > > >> >>> >> > The vram is allocated by qemu, see
> hw/vga.c:vga_common_init.
> > > >> >>> >> > qemu_ram_alloc under xen ends up calling xen_ram_alloc that
> calls
> > > >> >>> >> > xc_domain_populate_physmap_exact.
> > > >> >>> >> > xc_domain_populate_physmap_exact is the hypercall that
> should ask Xen to
> > > >> >>> >> > add the missing vram pages in the guest. Maybe this
> hypercall is failing
> > > >> >>> >> > in your case?
> > > >> >>> >>
> > > >> >>> >>
> > > >> >>> >> Hi,
> > > >> >>> >>
> > > >> >>> >>    I continue to invesgate this bug and find
> hypercall_mmu_update in
> > > >> >>> >> qemu_remap_bucket(xc_map_foreign_bulk) is failing:
> > > >> >>> >>
> > > >> >>> >> do_mmu_update
> > > >> >>> >>       ->mod_l1_entry
> > > >> >>> >>              ->  if ( !p2m_is_ram(p2mt) || unlikely(mfn ==
> INVALID_MFN) )
> > > >> >>> >>                          return -EINVAL;
> > > >> >>> >>
> > > >> >>> >>    mfn==INVALID_MFN, because :
> > > >> >>> >>
> > > >> >>> >> mod_l1_entry
> > > >> >>> >>       ->gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> l1e_get_pfn(nl1e), &p2mt));
> > > >> >>> >>               ->p2m->get_entry
> > > >> >>> >>                         ->p2m_gfn_to_mfn
> > > >> >>> >>                                -> if ( gfn >
> p2m->max_mapped_pfn )
> > > >> >>> >>                                    /* This pfn is higher than
> the
> > > >> >>> >> highest the p2m map currently holds */
> > > >> >>> >>                                    return _mfn(INVALID_MFN);
> > > >> >>> >>
> > > >> >>> >>    The p2m->max_mapped_pfn is usually 0xfffff. In our case,
> > > >> >>> >> mmu_update.val exceeds 0x8000000100000000.  Additionally, l1e
> =
> > > >> >>> >> l1e_from_intpte(mmu_update.val); gfn=l1e_get_pfn(l1e ).
> Therefore, gfn
> > > >> >>> >> will exceed 0xfffff.
> > > >> >>> >>
> > > >> >>> >>    In the case of minios based stubdom, the mmu_update.vals
> do not
> > > >> >>> >> exceed 0x8000000100000000. Next, I will invesgate why
> mmu_update.val
> > > >> >>> >> exceeds 0x8000000100000000.
> > > >> >>> >
> > > >> >>> > It looks like the address of the guest that qemu is trying to
> map is not
> > > >> >>> > valid.
> > > >> >>> > Make sure you are running a guest with less than 2GB of ram,
> otherwise
> > > >> >>> > you need the patch series that Anthony sent on Friday:
> > > >> >>> >
> > > >> >>> > http://marc.info/?l=qemu-devel&m=131074042905711&w=2
> > > >> >>>
> > > >> >>> Not this problem. I never alloc more than 2GB for the hvm guest.
> The
> > > >> >>> call stack in qemu is:
> > > >> >>>
> > > >> >>> qemu_get_ram_ptr
> > > >> >>>       ->qemu_map_cache(addr, 0, 1)
> > > >> >>>                  -> if (!entry->vaddr_base || entry->paddr_index
> !=
> > > >> >>> address_index ||
> > > >> >>>
> !test_bit(address_offset >>
> > > >> >>> XC_PAGE_SHIFT, entry->valid_mapping)) {
> > > >> >>>                            qemu_remap_bucket(entry, size ? :
> > > >> >>> MCACHE_BUCKET_SIZE, address_index);
> > > >> >>>                                  ->xc_map_foreign_bulk(xen_xc,
> > > >> >>> xen_domid, PROT_READ|PROT_WRITE,
> > > >> >>>
> > > >> >>>                 pfns, err, nb_pfn);
> > > >> >>>
> > > >> >>> The qemu tries to map pages from hvm guest(xen_domid) to linux
> > > >> >>> stubdom. But some hvm pages' pfns are larger than 0xfffff. So,
> in the
> > > >> >>> p2m_gfn_to_mfn, the judgement condition is
> valid:(p2m->max_mapped_pfn
> > > >> >>> = 0xfffff)
> > > >> >>>
> > > >> >>>     if ( gfn > p2m->max_mapped_pfn )
> > > >> >>>         /* This pfn is higher than the highest the p2m map
> currently holds */
> > > >> >>>         return _mfn(INVALID_MFN);
> > > >> >>>
> > > >> >>>  In minios stubdom case, the hvm pages' pfns do not exceed
> 0xfffff.
> > > >> >>> Maybe the address translation in linux stubdom cause this
> probem?
> > > >> >>
> > > >> >> Trying to map a pfn > 0xfffff is clearly a mistake if the guest's
> memory
> > > >> >> does not exceed 2G:
> > > >> >>
> > > >> >> 0xfffff * 4096 > 2G
> > > >> >>
> > > >> >>
> > > >> >>>  BTW, in minios stubdom case, there seems no hvmloader process.
> Is it
> > > >> >>> needed in linux stubdom?
> > > >> >>
> > > >> >> hvmloader is the first thing that runs within the guest, it is
> not a
> > > >> >> process in the stubdom or in dom0.
> > > >> >> It is required in both minios and linux stubdoms.
> > > >> >
> > > >> > Hi Stefano,
> > > >> >
> > > >> >      I patched these patches, but we still have the same problem.
> > > >> > However, I noticed the qemu_get_ram_ptr(s->vram_offset) in
> > > >> > vga_common_init function was also failed. Maybe this can explain
> the
> > > >> > previous problem, which happened in the phase of trying to remap
> > > >> > 0xc0000-0xc8fff of hvm guest into stubdom.
> > > >> >
> > > >> >     I have traced the process of qemu_get_ram_ptr(s->vram_offset)
> and
> > > >> > located the failure in p2m_gfn_to_mfn function:
> > > >> >
> > > >> >     pod_retry_l3:
> > > >> >        if ( (l3e_get_flags(*l3e) & _PAGE_PRESENT) == 0 )
> > > >> >        {
> > > >> >                 .....
> > > >> >                 return _mfn(INVALID_MFN);
> > > >> >        }
> > > >> >
> > > >> >     I will continue to analyze this failure.
> > > >> >
> > > >> >     Thanks!
> > > >> >
> > > >> > Jiageng Yu.
> > > >> >
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >>     I compared the two executions of vga_common_init function in
> dom0
> > > >> and linux based stubdom. The former succeeded and the later was
> > > >> failed. First, they have the same call stack:
> > > >>
> > > >> Dom0 & Stubdom
> > > >> _________________________________________________________
> > > >> vga_common_init
> > > >>      ->qemu_get_ram_ptr(s->vram_offset)
> > > >>            ->block->host = xen_map_block(block->offset,
> block->length);
> > > >>                  ->xc_map_foreign_bulk()
> > > >>                         ->linux_privcmd_map_foreign_bulk()
> > > >>                                ->xen_remap_domain_mfn_range()
> > > >>                                      ->HYPERVISOR_mmu_update()
> > > >> __________________________________________________________
> > > >>
> > > >> Xen
> > > >> __________________________________________________________
> > > >> do_mmu_update()
> > > >>    ->case MMU_PT_UPDATE_PRESERVE_AD:
> > > >>    ->case PGT_l1_page_table:
> > > >>    ->mod_l1_entry(va, l1e, mfn,cmd == MMU_PT_UPDATE_PRESERVE_AD, v,
> pg_owner);
> > > >>           ->mfn_x(gfn_to_mfn(p2m_get_hostp2m(pg_dom),
> > > >> l1e_get_pfn(nl1e), &p2mt));
> > > >>                  ->gfn_to_mfn_type_p2m()
> > > >>                         ->p2m->get_entry(p2m, gfn, t, &a, q);
> > > >>                                ->p2m_gfn_to_mfn(p2m,gfn,t,&a,q)
> > > >>                                       ->if ( (l3e_get_flags(*l3e) &
> > > >> _PAGE_PRESENT) == 0 )
> > > >>                                       ->    Error happens!
> > > >>
> > > >> The qemu in dom0 can find the l3e of hvm guest, but the qemu in
> linux
> > > >> stubdom cannot find the l3e. In my case, s->vram_offset=0x40000000,
> > > >> vga_ram_size=0x800000. Therefore, we are going to map hvm guest's
> > > >> address area(pfn:0x40000, size:8M) into linux stubdom's address
> space.
> > > >>
> > > >> In p2m_gfn_to_mfn function, p2m->domain->domain_id=hvm guest,
> > > >> gfn=0x40000, t=p2m_mmio_dm.
> > > >> mfn = pagetable_get_mfn(p2m_get_pagetable(p2m) = 0x10746e;
> > > >> map_domain_page(mfn_x(mfn)) is also success. However, after
> executing:
> > > >> l3e += ( (0x40000 << PAGE_SHIFT) >> L3_PAGETABLE_SHIFT)
> > > >> the l3e->l3 =0 , and the error happens.
> > > >>
> > > >> So, in linux stubdom, when we are going to map the specified hvm
> > > >> guest's address(pfn:0x40000, size:8M), we find these pages of hvm
> > > >> guest are not present. This is never happened in qemu of dom0. Could
> > > >> you give me some prompts to this problem?
> > > >
> > > >
> > > > It seems that you are trying to map pages that don't exist.
> > > > The pages in question should be allocated by:
> > > >
> > > > qemu_ram_alloc(NULL, "vga.vram", vga_ram_size)
> > > >    qemu_ram_alloc_from_ptr
> > > >        xen_ram_alloc
> > > >            xc_domain_populate_physmap_exact
> > > >
> > > > so I would add some printf and printk on this code path to find out
> if
> > > > xc_domain_populate_physmap_exact fails for some reasons.
> > >
> > > Hmm.. the linux stubdom kernel had a wrong p2m pair
> > > <gfn(0x40000),mfn(0x127bd2)> for some reason. But next, the
> > > xc_domain_populate_physmap_exact will setup the correct p2m pair
> > > <gfn(0x40000),mfn(0x896b7)>. However, the p2m pair in stubdom kernel
> > > has not been updated, because the fllowing access to 0x40000 still
> > > uses 0x127bd2.
> >
> > The p2m for the guest domain is only one in Xen, so I cannot understand
> > how it is possible that you get the old mfn value.
> > Also there shouldn't even be an old value because before
> > xc_domain_populate_physmap_exact pfn 0x40000 wasn't even allocated in
> > the guest yet.
> >
> > Make sure you are using the right domid in both calls
> > (xc_domain_populate_physmap_exact and xc_map_foreign_bulk), also make
> > sure that libxenlight calls xc_domain_set_target and xs_set_target for
> > the stubdom otherwise the stubdom is not going to be privileged enough
> > to allocate and map memory of the guest.
> >
> >
> > > I notice you have a patch: xen: modify kernel mappings corresponding
> > > to granted pages. I think maybe it could slove my problem.
> >
> > That patch fixes a different issue, related to grant table mappings.
> >
> >
> >
> > OK. That is my fault.
> >
> > The root cause of previous problem is that the backend drivers in qemu
> are not stopped. To confirm this root cause,
> > I try to erase the codes about stubdom in xen_be_init function of old
> qemu. The same problem appears. The following
> > patch is to fix this issue in upstream qemu.
> >
> > diff --git a/xen-all.c b/xen-all.c
> > index b73fc43..8f0645e 100644
> > --- a/xen-all.c
> > +++ b/xen-all.c
> > @@ -472,12 +479,23 @@ static void cpu_handle_ioreq(void *opaque)
> >  static void xenstore_record_dm_state(XenIOState *s, const char *state)
> >  {
> >      char path[50];
> > +#ifdef CONFIG_STUBDOM
> > +    s->xenstore = xs_daemon_open();
> > +    if (s->xenstore == NULL) {
> > +        perror("xen: xenstore open");
> > +        return -errno;
> > +    }
> > +#endif
> >      snprintf(path, sizeof (path),
> "/local/domain/0/device-model/%u/state", xen_domid);
> >      if (!xs_write(s->xenstore, XBT_NULL, path, state, strlen(state))) {
> >          fprintf(stderr, "error recording dm state\n");
> >          exit(1);
> >      }
> > +#ifdef CONFIG_STUBDOM
> > +    xs_daemon_close(s->xenstore);
> > +    s->xenstore = NULL;
> > +#endif
> >  }
>
> Why do you need to re-open the xenstore connection here?
> It should be already been opened by xen_hvm_init, like in the normal
> case.
>
>
> >  static void xen_main_loop_prepare(XenIOState *state)
> > @@ -538,6 +556,7 @@ int xen_hvm_init(void)
> >
> >      state = qemu_mallocz(sizeof (XenIOState));
> >
> > +#ifndef CONFIG_STUBDOM
> >      state->xce_handle = xen_xc_evtchn_open(NULL, 0);
> >      if (state->xce_handle == XC_HANDLER_INITIAL_VALUE) {
> >          perror("xen: event channel open");
> > @@ -549,6 +568,10 @@ int xen_hvm_init(void)
> >          perror("xen: xenstore open");
> >          return -errno;
> >      }
> > +#else
> > +    state->xce_handle = XC_HANDLER_INITIAL_VALUE;
> > +    state->xenstore = NULL;
> > +#endif
> >
>
> So you are explicitly avoiding to open the xenstore connection from
> xen_hvm_init, why?
> I think you might be trying to fix a race condition, maybe something is
> not ready yet at this point that becomes ready later?
>
>
> >      state->exit.notify = xen_exit_notifier;
> >      qemu_add_exit_notifier(&state->exit);
> > @@ -575,9 +598,10 @@ int xen_hvm_init(void)
> >
> >      state->ioreq_local_port = qemu_mallocz(smp_cpus * sizeof
> (evtchn_port_t));
> >
> >      /* FIXME: how about if we overflow the page here? */
> >      for (i = 0; i < smp_cpus; i++) {
> > -        rc = xc_evtchn_bind_interdomain(state->xce_handle, xen_domid,
> > +       rc = xc_evtchn_bind_interdomain(xen_xc, xen_domid,
> >
> xen_vcpu_eport(state->shared_page, i));
> >          if (rc == -1) {
> >              fprintf(stderr, "bind interdomain ioctl error %d\n", errno);
> >
>
> This cannot be right: xc_evtchn_bind_interdomain takes a xc_evtchn* as
> first paramter while xen_xc is xc_interface*
> This change would prevent you from receiving any IO request
> notifications from Xen.
>
>
> > The new problem is my stubdom hangs at:
> >
> > hvmloader:
> >      ->main()
> >             ->pci_setup()
> >                     ->pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);
> >
> > I am investigating this problem. The pci_writeb will finally call the
> hvm_set_pci_link_route in Xen:
> >
> > hvmloader:             pci_writeb(PCI_ISA_DEVFN, 0x60 + link, isa_irq);
> > qemu(stubdom):     PCIHostState->data_handler->write()
> > qemu(stubdom):     i440fx_write_config_xen()
> > qemu(stubdom):     xen_piix_pci_write_config_client()
> > xenctrl:                   xc_hvm_set_pci_link_route()
> >
> > The ioport is registered by pci_host_data_register_ioport(0xcfc, s)
> function.
> >
> > I will find out why not invoke i440fx_write_config_xen() in my case. I
> will also read the pciutils.patch of minios stubdom
> > and maybe find something interesting.
>
> I think you are not receiving any IO request notifications from Xen
> because of the previous change.
> It is probably worth adding a printf into xen-all.c:handle_ioreq to see
> if you receive something.



I have noticed the mistake. In fact, we shall stop the map_foreign_pages of
xen_console and xenfb devices in qemu. Because the front drivers in stubdom
has already map the memories. This is my new patch. But it is not stable, I
am testing it.

We use xc_handle to map foreign pages in xenfb and xen_console devices. If
qemu running on stubdom, the xc_handle is invalid.


diff --git a/hw/xen_backend.c b/hw/xen_backend.c
index cfb53c8..11c53fe 100644
--- a/hw/xen_backend.c
+++ b/hw/xen_backend.c
@@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
 XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
 struct xs_handle *xenstore = NULL;
 const char *xen_protocol;
+XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;

 /* private */
 static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
QTAILQ_HEAD_INITIALIZER(xendevs);
@@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)

 int xen_be_init(void)
 {
+#ifndef CONFIG_STUBDOM
     xenstore = xs_daemon_open();
     if (!xenstore) {
         xen_be_printf(NULL, 0, "can't connect to xenstored\n");
@@ -665,10 +667,16 @@ int xen_be_init(void)
         goto err;
     }

+    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
+        goto err;
+    }
+#endif
+
     if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
         /* Check if xen_init() have been called */
         goto err;
     }
+
     return 0;

 err:
diff --git a/hw/xen_backend.h b/hw/xen_backend.h
index 9d36df3..bc5a157 100644
--- a/hw/xen_backend.h
+++ b/hw/xen_backend.h
@@ -59,6 +59,9 @@ extern XenXC xen_xc;
 extern struct xs_handle *xenstore;
 extern const char *xen_protocol;

+/* invalid in linux stubdom */
+extern XenXC xc_handle;
+
 /* xenstore helper functions */
 int xenstore_write_str(const char *base, const char *node, const char
*val);
 int xenstore_write_int(const char *base, const char *node, int ival);
diff --git a/hw/xen_console.c b/hw/xen_console.c
index c6c8163..66b6dd7 100644
--- a/hw/xen_console.c
+++ b/hw/xen_console.c
@@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
     if (xenstore_read_int(con->console, "limit", &limit) == 0)
  con->buffer.max_capacity = limit;

-    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
+   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
           XC_PAGE_SIZE,
           PROT_READ|PROT_WRITE,
           con->ring_ref);
diff --git a/hw/xenfb.c b/hw/xenfb.c
index 039076a..278fa60 100644
--- a/hw/xenfb.c
+++ b/hw/xenfb.c
@@ -104,7 +104,7 @@ static int common_bind(struct common *c)
     if (xenstore_read_fe_int(&c->xendev, "event-channel",
&c->xendev.remote_port) == -1)
  return -1;

-    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
+   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
        XC_PAGE_SIZE,
        PROT_READ | PROT_WRITE, mfn);
     if (c->page == NULL)
@@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
     fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);

     xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
-    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
+   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
           PROT_READ, pgmfns, n_fbdirs);
     if (map == NULL)
  goto out;
     xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
     munmap(map, n_fbdirs * XC_PAGE_SIZE);

-    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
+   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
       PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
     if (xenfb->pixels == NULL)
  goto out;
diff --git a/xen-all.c b/xen-all.c
index b73fc43..04dfb51 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -527,12 +534,22 @@ int xen_init(void)
         return -1;
     }

+#ifdef CONFIG_STUBDOM
+    return 0;
+#endif
+
+    xc_handle = xen_xc_interface_open(0, 0, 0);
+    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
+        xen_be_printf(NULL, 0, "can't open xen interface\n");
+        return -1;
+    }
+
     return 0;
 }

[-- Attachment #1.2: Type: text/html, Size: 31611 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5302 bytes --]

On Fri, 29 Jul 2011, Jiageng Yu wrote:
> I have noticed the mistake. In fact, we shall stop the map_foreign_pages of xen_console and xenfb devices in qemu. Because
> the front drivers in stubdom has already map the memories. This is my new patch. But it is not stable, I am testing it.
>  
> We use xc_handle to map foreign pages in xenfb and xen_console devices. If qemu running on stubdom, the xc_handle is
> invalid.
>  
>  
> diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> index cfb53c8..11c53fe 100644
> --- a/hw/xen_backend.c
> +++ b/hw/xen_backend.c
> @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
>  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>  struct xs_handle *xenstore = NULL;
>  const char *xen_protocol;
> +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
>  
>  /* private */
>  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs = QTAILQ_HEAD_INITIALIZER(xendevs);
> @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
>  
>  int xen_be_init(void)
>  {
> +#ifndef CONFIG_STUBDOM
>      xenstore = xs_daemon_open();
>      if (!xenstore) {
>          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
> @@ -665,10 +667,16 @@ int xen_be_init(void)
>          goto err;
>      }
>  
> +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> +        goto err;
> +    }
> +#endif
> +
>      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
>          /* Check if xen_init() have been called */
>          goto err;
>      }
> +
>      return 0;
>  
>  err:
> diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> index 9d36df3..bc5a157 100644
> --- a/hw/xen_backend.h
> +++ b/hw/xen_backend.h
> @@ -59,6 +59,9 @@ extern XenXC xen_xc;
>  extern struct xs_handle *xenstore;
>  extern const char *xen_protocol;
>  
> +/* invalid in linux stubdom */
> +extern XenXC xc_handle;
> +
>  /* xenstore helper functions */
>  int xenstore_write_str(const char *base, const char *node, const char *val);
>  int xenstore_write_int(const char *base, const char *node, int ival);
> diff --git a/hw/xen_console.c b/hw/xen_console.c
> index c6c8163..66b6dd7 100644
> --- a/hw/xen_console.c
> +++ b/hw/xen_console.c
> @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
>      if (xenstore_read_int(con->console, "limit", &limit) == 0)
>   con->buffer.max_capacity = limit;
>  
> -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
> +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
>            XC_PAGE_SIZE,
>            PROT_READ|PROT_WRITE,
>            con->ring_ref);
> diff --git a/hw/xenfb.c b/hw/xenfb.c
> index 039076a..278fa60 100644
> --- a/hw/xenfb.c
> +++ b/hw/xenfb.c
> @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
>      if (xenstore_read_fe_int(&c->xendev, "event-channel", &c->xendev.remote_port) == -1)
>   return -1;
>  
> -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
> +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
>         XC_PAGE_SIZE,
>         PROT_READ | PROT_WRITE, mfn);
>      if (c->page == NULL)
> @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
>      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
>  
>      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
> -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>            PROT_READ, pgmfns, n_fbdirs);
>      if (map == NULL)
>   goto out;
>      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
>      munmap(map, n_fbdirs * XC_PAGE_SIZE);
>  
> -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
>      if (xenfb->pixels == NULL)
>   goto out;
> diff --git a/xen-all.c b/xen-all.c
> index b73fc43..04dfb51 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -527,12 +534,22 @@ int xen_init(void)
>          return -1;
>      }
>  
> +#ifdef CONFIG_STUBDOM
> +    return 0;
> +#endif
> +
> +    xc_handle = xen_xc_interface_open(0, 0, 0);
> +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> +        xen_be_printf(NULL, 0, "can't open xen interface\n");
> +        return -1;
> +    }
> +
>      return 0;
>  }
> 
> 

I think that the backends shouldn't be initialized at all when running
in the stubdom, so I would do something like this instead:

diff --git a/xen-all.c b/xen-all.c
index 167bed6..e3f630b 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -922,6 +922,7 @@ int xen_hvm_init(void)
     cpu_register_phys_memory_client(&state->client);
     state->log_for_dirtybit = NULL;
 
+#ifndef CONFIG_STUBDOM
     /* Initialize backend core & drivers */
     if (xen_be_init() != 0) {
         fprintf(stderr, "%s: xen backend core setup failed\n", __FUNCTION__);
@@ -930,7 +931,7 @@ int xen_hvm_init(void)
     xen_be_register("console", &xen_console_ops);
     xen_be_register("vkbd", &xen_kbdmouse_ops);
     xen_be_register("qdisk", &xen_blkdev_ops);
-
+#endif
     return 0;
 }
 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1.1: Type: text/plain, Size: 5529 bytes --]

2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>

> On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > I have noticed the mistake. In fact, we shall stop the map_foreign_pages
> of xen_console and xenfb devices in qemu. Because
> > the front drivers in stubdom has already map the memories. This is my new
> patch. But it is not stable, I am testing it.
> >
> > We use xc_handle to map foreign pages in xenfb and xen_console devices.
> If qemu running on stubdom, the xc_handle is
> > invalid.
> >
> >
> > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> > index cfb53c8..11c53fe 100644
> > --- a/hw/xen_backend.c
> > +++ b/hw/xen_backend.c
> > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
> >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
> >  struct xs_handle *xenstore = NULL;
> >  const char *xen_protocol;
> > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
> >
> >  /* private */
> >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
> QTAILQ_HEAD_INITIALIZER(xendevs);
> > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
> >
> >  int xen_be_init(void)
> >  {
> > +#ifndef CONFIG_STUBDOM
> >      xenstore = xs_daemon_open();
> >      if (!xenstore) {
> >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
> > @@ -665,10 +667,16 @@ int xen_be_init(void)
> >          goto err;
> >      }
> >
> > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > +        goto err;
> > +    }
> > +#endif
> > +
> >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
> >          /* Check if xen_init() have been called */
> >          goto err;
> >      }
> > +
> >      return 0;
> >
> >  err:
> > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> > index 9d36df3..bc5a157 100644
> > --- a/hw/xen_backend.h
> > +++ b/hw/xen_backend.h
> > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
> >  extern struct xs_handle *xenstore;
> >  extern const char *xen_protocol;
> >
> > +/* invalid in linux stubdom */
> > +extern XenXC xc_handle;
> > +
> >  /* xenstore helper functions */
> >  int xenstore_write_str(const char *base, const char *node, const char
> *val);
> >  int xenstore_write_int(const char *base, const char *node, int ival);
> > diff --git a/hw/xen_console.c b/hw/xen_console.c
> > index c6c8163..66b6dd7 100644
> > --- a/hw/xen_console.c
> > +++ b/hw/xen_console.c
> > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
> >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
> >   con->buffer.max_capacity = limit;
> >
> > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
> > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
> >            XC_PAGE_SIZE,
> >            PROT_READ|PROT_WRITE,
> >            con->ring_ref);
> > diff --git a/hw/xenfb.c b/hw/xenfb.c
> > index 039076a..278fa60 100644
> > --- a/hw/xenfb.c
> > +++ b/hw/xenfb.c
> > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
> >      if (xenstore_read_fe_int(&c->xendev, "event-channel",
> &c->xendev.remote_port) == -1)
> >   return -1;
> >
> > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
> > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
> >         XC_PAGE_SIZE,
> >         PROT_READ | PROT_WRITE, mfn);
> >      if (c->page == NULL)
> > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
> >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
> >
> >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
> > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> >            PROT_READ, pgmfns, n_fbdirs);
> >      if (map == NULL)
> >   goto out;
> >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
> >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
> >
> > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
> >      if (xenfb->pixels == NULL)
> >   goto out;
> > diff --git a/xen-all.c b/xen-all.c
> > index b73fc43..04dfb51 100644
> > --- a/xen-all.c
> > +++ b/xen-all.c
> > @@ -527,12 +534,22 @@ int xen_init(void)
> >          return -1;
> >      }
> >
> > +#ifdef CONFIG_STUBDOM
> > +    return 0;
> > +#endif
> > +
> > +    xc_handle = xen_xc_interface_open(0, 0, 0);
> > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
> > +        return -1;
> > +    }
> > +
> >      return 0;
> >  }
> >
> >
>
> I think that the backends shouldn't be initialized at all when running
> in the stubdom, so I would do something like this instead:
>
> diff --git a/xen-all.c b/xen-all.c
> index 167bed6..e3f630b 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -922,6 +922,7 @@ int xen_hvm_init(void)
>     cpu_register_phys_memory_client(&state->client);
>     state->log_for_dirtybit = NULL;
>
> +#ifndef CONFIG_STUBDOM
>     /* Initialize backend core & drivers */
>     if (xen_be_init() != 0) {
>         fprintf(stderr, "%s: xen backend core setup failed\n",
> __FUNCTION__);
> @@ -930,7 +931,7 @@ int xen_hvm_init(void)
>     xen_be_register("console", &xen_console_ops);
>     xen_be_register("vkbd", &xen_kbdmouse_ops);
>     xen_be_register("qdisk", &xen_blkdev_ops);
> -
> +#endif
>     return 0;
>  }
>



Yes. this implementation of stubdom is more closer to the old qemu's
implementation. Which branch this patch works on? The qemu-dm-v15 would not
so lucky.

[-- Attachment #1.2: Type: text/html, Size: 7288 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 6006 bytes --]

On Fri, 29 Jul 2011, Jiageng Yu wrote:
> 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > I have noticed the mistake. In fact, we shall stop the map_foreign_pages of xen_console and xenfb devices in qemu.
> Because
> > the front drivers in stubdom has already map the memories. This is my new patch. But it is not stable, I am
> testing it.
> >  
> > We use xc_handle to map foreign pages in xenfb and xen_console devices. If qemu running on stubdom, the xc_handle
> is
> > invalid.
> >  
> >  
> > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> > index cfb53c8..11c53fe 100644
> > --- a/hw/xen_backend.c
> > +++ b/hw/xen_backend.c
> > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
> >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
> >  struct xs_handle *xenstore = NULL;
> >  const char *xen_protocol;
> > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
> >  
> >  /* private */
> >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs = QTAILQ_HEAD_INITIALIZER(xendevs);
> > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
> >  
> >  int xen_be_init(void)
> >  {
> > +#ifndef CONFIG_STUBDOM
> >      xenstore = xs_daemon_open();
> >      if (!xenstore) {
> >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
> > @@ -665,10 +667,16 @@ int xen_be_init(void)
> >          goto err;
> >      }
> >  
> > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > +        goto err;
> > +    }
> > +#endif
> > +
> >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
> >          /* Check if xen_init() have been called */
> >          goto err;
> >      }
> > +
> >      return 0;
> >  
> >  err:
> > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> > index 9d36df3..bc5a157 100644
> > --- a/hw/xen_backend.h
> > +++ b/hw/xen_backend.h
> > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
> >  extern struct xs_handle *xenstore;
> >  extern const char *xen_protocol;
> >  
> > +/* invalid in linux stubdom */
> > +extern XenXC xc_handle;
> > +
> >  /* xenstore helper functions */
> >  int xenstore_write_str(const char *base, const char *node, const char *val);
> >  int xenstore_write_int(const char *base, const char *node, int ival);
> > diff --git a/hw/xen_console.c b/hw/xen_console.c
> > index c6c8163..66b6dd7 100644
> > --- a/hw/xen_console.c
> > +++ b/hw/xen_console.c
> > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
> >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
> >   con->buffer.max_capacity = limit;
> >  
> > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
> > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
> >            XC_PAGE_SIZE,
> >            PROT_READ|PROT_WRITE,
> >            con->ring_ref);
> > diff --git a/hw/xenfb.c b/hw/xenfb.c
> > index 039076a..278fa60 100644
> > --- a/hw/xenfb.c
> > +++ b/hw/xenfb.c
> > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
> >      if (xenstore_read_fe_int(&c->xendev, "event-channel", &c->xendev.remote_port) == -1)
> >   return -1;
> >  
> > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
> > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
> >         XC_PAGE_SIZE,
> >         PROT_READ | PROT_WRITE, mfn);
> >      if (c->page == NULL)
> > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
> >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
> >  
> >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
> > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> >            PROT_READ, pgmfns, n_fbdirs);
> >      if (map == NULL)
> >   goto out;
> >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
> >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
> >  
> > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
> >      if (xenfb->pixels == NULL)
> >   goto out;
> > diff --git a/xen-all.c b/xen-all.c
> > index b73fc43..04dfb51 100644
> > --- a/xen-all.c
> > +++ b/xen-all.c
> > @@ -527,12 +534,22 @@ int xen_init(void)
> >          return -1;
> >      }
> >  
> > +#ifdef CONFIG_STUBDOM
> > +    return 0;
> > +#endif
> > +
> > +    xc_handle = xen_xc_interface_open(0, 0, 0);
> > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
> > +        return -1;
> > +    }
> > +
> >      return 0;
> >  }
> >
> >
> 
> I think that the backends shouldn't be initialized at all when running
> in the stubdom, so I would do something like this instead:
> 
> diff --git a/xen-all.c b/xen-all.c
> index 167bed6..e3f630b 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -922,6 +922,7 @@ int xen_hvm_init(void)
>     cpu_register_phys_memory_client(&state->client);
>     state->log_for_dirtybit = NULL;
> 
> +#ifndef CONFIG_STUBDOM
>     /* Initialize backend core & drivers */
>     if (xen_be_init() != 0) {
>         fprintf(stderr, "%s: xen backend core setup failed\n", __FUNCTION__);
> @@ -930,7 +931,7 @@ int xen_hvm_init(void)
>     xen_be_register("console", &xen_console_ops);
>     xen_be_register("vkbd", &xen_kbdmouse_ops);
>     xen_be_register("qdisk", &xen_blkdev_ops);
> -
> +#endif
>     return 0;
>  }
>  
> 
> 
>  
> Yes. this implementation of stubdom is more closer to the old qemu's implementation. Which branch this patch works on? The
> qemu-dm-v15 would not so lucky.

That branch is old now, checkout this one:

git://xenbits.xen.org/people/sstabellini/qemu-dm.git xen-stable-0.15

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1.1: Type: text/plain, Size: 6143 bytes --]

2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>

> On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > > I have noticed the mistake. In fact, we shall stop the
> map_foreign_pages of xen_console and xenfb devices in qemu.
> > Because
> > > the front drivers in stubdom has already map the memories. This is my
> new patch. But it is not stable, I am
> > testing it.
> > >
> > > We use xc_handle to map foreign pages in xenfb and xen_console devices.
> If qemu running on stubdom, the xc_handle
> > is
> > > invalid.
> > >
> > >
> > > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> > > index cfb53c8..11c53fe 100644
> > > --- a/hw/xen_backend.c
> > > +++ b/hw/xen_backend.c
> > > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
> > >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
> > >  struct xs_handle *xenstore = NULL;
> > >  const char *xen_protocol;
> > > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
> > >
> > >  /* private */
> > >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
> QTAILQ_HEAD_INITIALIZER(xendevs);
> > > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
> > >
> > >  int xen_be_init(void)
> > >  {
> > > +#ifndef CONFIG_STUBDOM
> > >      xenstore = xs_daemon_open();
> > >      if (!xenstore) {
> > >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
> > > @@ -665,10 +667,16 @@ int xen_be_init(void)
> > >          goto err;
> > >      }
> > >
> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > > +        goto err;
> > > +    }
> > > +#endif
> > > +
> > >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
> > >          /* Check if xen_init() have been called */
> > >          goto err;
> > >      }
> > > +
> > >      return 0;
> > >
> > >  err:
> > > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> > > index 9d36df3..bc5a157 100644
> > > --- a/hw/xen_backend.h
> > > +++ b/hw/xen_backend.h
> > > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
> > >  extern struct xs_handle *xenstore;
> > >  extern const char *xen_protocol;
> > >
> > > +/* invalid in linux stubdom */
> > > +extern XenXC xc_handle;
> > > +
> > >  /* xenstore helper functions */
> > >  int xenstore_write_str(const char *base, const char *node, const char
> *val);
> > >  int xenstore_write_int(const char *base, const char *node, int ival);
> > > diff --git a/hw/xen_console.c b/hw/xen_console.c
> > > index c6c8163..66b6dd7 100644
> > > --- a/hw/xen_console.c
> > > +++ b/hw/xen_console.c
> > > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
> > >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
> > >   con->buffer.max_capacity = limit;
> > >
> > > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
> > > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
> > >            XC_PAGE_SIZE,
> > >            PROT_READ|PROT_WRITE,
> > >            con->ring_ref);
> > > diff --git a/hw/xenfb.c b/hw/xenfb.c
> > > index 039076a..278fa60 100644
> > > --- a/hw/xenfb.c
> > > +++ b/hw/xenfb.c
> > > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
> > >      if (xenstore_read_fe_int(&c->xendev, "event-channel",
> &c->xendev.remote_port) == -1)
> > >   return -1;
> > >
> > > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
> > > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
> > >         XC_PAGE_SIZE,
> > >         PROT_READ | PROT_WRITE, mfn);
> > >      if (c->page == NULL)
> > > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
> > >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
> > >
> > >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
> > > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> > >            PROT_READ, pgmfns, n_fbdirs);
> > >      if (map == NULL)
> > >   goto out;
> > >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
> > >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
> > >
> > > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > > +   xenfb->pixels = xc_map_foreign_pages(xc_handle,
> xenfb->c.xendev.dom,
> > >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
> > >      if (xenfb->pixels == NULL)
> > >   goto out;
> > > diff --git a/xen-all.c b/xen-all.c
> > > index b73fc43..04dfb51 100644
> > > --- a/xen-all.c
> > > +++ b/xen-all.c
> > > @@ -527,12 +534,22 @@ int xen_init(void)
> > >          return -1;
> > >      }
> > >
> > > +#ifdef CONFIG_STUBDOM
> > > +    return 0;
> > > +#endif
> > > +
> > > +    xc_handle = xen_xc_interface_open(0, 0, 0);
> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
> > > +        return -1;
> > > +    }
> > > +
> > >      return 0;
> > >  }
> > >
> > >
> >
> > I think that the backends shouldn't be initialized at all when running
> > in the stubdom, so I would do something like this instead:
> >
> > diff --git a/xen-all.c b/xen-all.c
> > index 167bed6..e3f630b 100644
> > --- a/xen-all.c
> > +++ b/xen-all.c
> > @@ -922,6 +922,7 @@ int xen_hvm_init(void)
> >     cpu_register_phys_memory_client(&state->client);
> >     state->log_for_dirtybit = NULL;
> >
> > +#ifndef CONFIG_STUBDOM
> >     /* Initialize backend core & drivers */
> >     if (xen_be_init() != 0) {
> >         fprintf(stderr, "%s: xen backend core setup failed\n",
> __FUNCTION__);
> > @@ -930,7 +931,7 @@ int xen_hvm_init(void)
> >     xen_be_register("console", &xen_console_ops);
> >     xen_be_register("vkbd", &xen_kbdmouse_ops);
> >     xen_be_register("qdisk", &xen_blkdev_ops);
> > -
> > +#endif
> >     return 0;
> >  }
> >
> >
> >
> >
> > Yes. this implementation of stubdom is more closer to the old qemu's
> implementation. Which branch this patch works on? The
> > qemu-dm-v15 would not so lucky.
>
> That branch is old now, checkout this one:
>
> git://xenbits.xen.org/people/sstabellini/qemu-dm.git xen-stable-0.15



Do you mean I migrate my work to this qemu branch?

[-- Attachment #1.2: Type: text/html, Size: 8508 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 6575 bytes --]

On Fri, 29 Jul 2011, Jiageng Yu wrote:
> 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > On Fri, 29 Jul 2011, Jiageng Yu wrote:
> > > I have noticed the mistake. In fact, we shall stop the map_foreign_pages of xen_console and xenfb devices in
> qemu.
> > Because
> > > the front drivers in stubdom has already map the memories. This is my new patch. But it is not stable, I am
> > testing it.
> > >  
> > > We use xc_handle to map foreign pages in xenfb and xen_console devices. If qemu running on stubdom, the
> xc_handle
> > is
> > > invalid.
> > >  
> > >  
> > > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> > > index cfb53c8..11c53fe 100644
> > > --- a/hw/xen_backend.c
> > > +++ b/hw/xen_backend.c
> > > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
> > >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
> > >  struct xs_handle *xenstore = NULL;
> > >  const char *xen_protocol;
> > > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
> > >  
> > >  /* private */
> > >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs = QTAILQ_HEAD_INITIALIZER(xendevs);
> > > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
> > >  
> > >  int xen_be_init(void)
> > >  {
> > > +#ifndef CONFIG_STUBDOM
> > >      xenstore = xs_daemon_open();
> > >      if (!xenstore) {
> > >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
> > > @@ -665,10 +667,16 @@ int xen_be_init(void)
> > >          goto err;
> > >      }
> > >  
> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > > +        goto err;
> > > +    }
> > > +#endif
> > > +
> > >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
> > >          /* Check if xen_init() have been called */
> > >          goto err;
> > >      }
> > > +
> > >      return 0;
> > >  
> > >  err:
> > > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> > > index 9d36df3..bc5a157 100644
> > > --- a/hw/xen_backend.h
> > > +++ b/hw/xen_backend.h
> > > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
> > >  extern struct xs_handle *xenstore;
> > >  extern const char *xen_protocol;
> > >  
> > > +/* invalid in linux stubdom */
> > > +extern XenXC xc_handle;
> > > +
> > >  /* xenstore helper functions */
> > >  int xenstore_write_str(const char *base, const char *node, const char *val);
> > >  int xenstore_write_int(const char *base, const char *node, int ival);
> > > diff --git a/hw/xen_console.c b/hw/xen_console.c
> > > index c6c8163..66b6dd7 100644
> > > --- a/hw/xen_console.c
> > > +++ b/hw/xen_console.c
> > > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
> > >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
> > >   con->buffer.max_capacity = limit;
> > >  
> > > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
> > > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
> > >            XC_PAGE_SIZE,
> > >            PROT_READ|PROT_WRITE,
> > >            con->ring_ref);
> > > diff --git a/hw/xenfb.c b/hw/xenfb.c
> > > index 039076a..278fa60 100644
> > > --- a/hw/xenfb.c
> > > +++ b/hw/xenfb.c
> > > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
> > >      if (xenstore_read_fe_int(&c->xendev, "event-channel", &c->xendev.remote_port) == -1)
> > >   return -1;
> > >  
> > > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
> > > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
> > >         XC_PAGE_SIZE,
> > >         PROT_READ | PROT_WRITE, mfn);
> > >      if (c->page == NULL)
> > > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
> > >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
> > >  
> > >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
> > > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> > >            PROT_READ, pgmfns, n_fbdirs);
> > >      if (map == NULL)
> > >   goto out;
> > >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
> > >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
> > >  
> > > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
> > > +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
> > >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
> > >      if (xenfb->pixels == NULL)
> > >   goto out;
> > > diff --git a/xen-all.c b/xen-all.c
> > > index b73fc43..04dfb51 100644
> > > --- a/xen-all.c
> > > +++ b/xen-all.c
> > > @@ -527,12 +534,22 @@ int xen_init(void)
> > >          return -1;
> > >      }
> > >  
> > > +#ifdef CONFIG_STUBDOM
> > > +    return 0;
> > > +#endif
> > > +
> > > +    xc_handle = xen_xc_interface_open(0, 0, 0);
> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
> > > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
> > > +        return -1;
> > > +    }
> > > +
> > >      return 0;
> > >  }
> > >
> > >
> >
> > I think that the backends shouldn't be initialized at all when running
> > in the stubdom, so I would do something like this instead:
> >
> > diff --git a/xen-all.c b/xen-all.c
> > index 167bed6..e3f630b 100644
> > --- a/xen-all.c
> > +++ b/xen-all.c
> > @@ -922,6 +922,7 @@ int xen_hvm_init(void)
> >     cpu_register_phys_memory_client(&state->client);
> >     state->log_for_dirtybit = NULL;
> >
> > +#ifndef CONFIG_STUBDOM
> >     /* Initialize backend core & drivers */
> >     if (xen_be_init() != 0) {
> >         fprintf(stderr, "%s: xen backend core setup failed\n", __FUNCTION__);
> > @@ -930,7 +931,7 @@ int xen_hvm_init(void)
> >     xen_be_register("console", &xen_console_ops);
> >     xen_be_register("vkbd", &xen_kbdmouse_ops);
> >     xen_be_register("qdisk", &xen_blkdev_ops);
> > -
> > +#endif
> >     return 0;
> >  }
> >  
> >
> >
> >  
> > Yes. this implementation of stubdom is more closer to the old qemu's implementation. Which branch this patch works
> on? The
> > qemu-dm-v15 would not so lucky.
> 
> That branch is old now, checkout this one:
> 
> git://xenbits.xen.org/people/sstabellini/qemu-dm.git xen-stable-0.15
> 
> 
>  
> Do you mean I migrate my work to this qemu branch?
 
Yes, if it doesn't cost too much time for you to rebase your patches.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Fri, 29 Jul 2011, Jiageng Yu wrote:
>> 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>> On Fri, 29 Jul 2011, Jiageng Yu wrote:
>> > 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>> > On Fri, 29 Jul 2011, Jiageng Yu wrote:
>> > > I have noticed the mistake. In fact, we shall stop the map_foreign_pages of xen_console and xenfb devices in
>> qemu.
>> > Because
>> > > the front drivers in stubdom has already map the memories. This is my new patch. But it is not stable, I am
>> > testing it.
>> > >
>> > > We use xc_handle to map foreign pages in xenfb and xen_console devices. If qemu running on stubdom, the
>> xc_handle
>> > is
>> > > invalid.
>> > >
>> > >
>> > > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
>> > > index cfb53c8..11c53fe 100644
>> > > --- a/hw/xen_backend.c
>> > > +++ b/hw/xen_backend.c
>> > > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
>> > >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>> > >  struct xs_handle *xenstore = NULL;
>> > >  const char *xen_protocol;
>> > > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
>> > >
>> > >  /* private */
>> > >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs = QTAILQ_HEAD_INITIALIZER(xendevs);
>> > > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
>> > >
>> > >  int xen_be_init(void)
>> > >  {
>> > > +#ifndef CONFIG_STUBDOM
>> > >      xenstore = xs_daemon_open();
>> > >      if (!xenstore) {
>> > >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
>> > > @@ -665,10 +667,16 @@ int xen_be_init(void)
>> > >          goto err;
>> > >      }
>> > >
>> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
>> > > +        goto err;
>> > > +    }
>> > > +#endif
>> > > +
>> > >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
>> > >          /* Check if xen_init() have been called */
>> > >          goto err;
>> > >      }
>> > > +
>> > >      return 0;
>> > >
>> > >  err:
>> > > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
>> > > index 9d36df3..bc5a157 100644
>> > > --- a/hw/xen_backend.h
>> > > +++ b/hw/xen_backend.h
>> > > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
>> > >  extern struct xs_handle *xenstore;
>> > >  extern const char *xen_protocol;
>> > >
>> > > +/* invalid in linux stubdom */
>> > > +extern XenXC xc_handle;
>> > > +
>> > >  /* xenstore helper functions */
>> > >  int xenstore_write_str(const char *base, const char *node, const char *val);
>> > >  int xenstore_write_int(const char *base, const char *node, int ival);
>> > > diff --git a/hw/xen_console.c b/hw/xen_console.c
>> > > index c6c8163..66b6dd7 100644
>> > > --- a/hw/xen_console.c
>> > > +++ b/hw/xen_console.c
>> > > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
>> > >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
>> > >   con->buffer.max_capacity = limit;
>> > >
>> > > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
>> > > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
>> > >            XC_PAGE_SIZE,
>> > >            PROT_READ|PROT_WRITE,
>> > >            con->ring_ref);
>> > > diff --git a/hw/xenfb.c b/hw/xenfb.c
>> > > index 039076a..278fa60 100644
>> > > --- a/hw/xenfb.c
>> > > +++ b/hw/xenfb.c
>> > > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
>> > >      if (xenstore_read_fe_int(&c->xendev, "event-channel", &c->xendev.remote_port) == -1)
>> > >   return -1;
>> > >
>> > > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
>> > > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
>> > >         XC_PAGE_SIZE,
>> > >         PROT_READ | PROT_WRITE, mfn);
>> > >      if (c->page == NULL)
>> > > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
>> > >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
>> > >
>> > >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
>> > > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
>> > > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>> > >            PROT_READ, pgmfns, n_fbdirs);
>> > >      if (map == NULL)
>> > >   goto out;
>> > >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
>> > >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
>> > >
>> > > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
>> > > +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>> > >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
>> > >      if (xenfb->pixels == NULL)
>> > >   goto out;
>> > > diff --git a/xen-all.c b/xen-all.c
>> > > index b73fc43..04dfb51 100644
>> > > --- a/xen-all.c
>> > > +++ b/xen-all.c
>> > > @@ -527,12 +534,22 @@ int xen_init(void)
>> > >          return -1;
>> > >      }
>> > >
>> > > +#ifdef CONFIG_STUBDOM
>> > > +    return 0;
>> > > +#endif
>> > > +
>> > > +    xc_handle = xen_xc_interface_open(0, 0, 0);
>> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
>> > > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
>> > > +        return -1;
>> > > +    }
>> > > +
>> > >      return 0;
>> > >  }
>> > >
>> > >
>> >
>> > I think that the backends shouldn't be initialized at all when running
>> > in the stubdom, so I would do something like this instead:
>> >
>> > diff --git a/xen-all.c b/xen-all.c
>> > index 167bed6..e3f630b 100644
>> > --- a/xen-all.c
>> > +++ b/xen-all.c
>> > @@ -922,6 +922,7 @@ int xen_hvm_init(void)
>> >     cpu_register_phys_memory_client(&state->client);
>> >     state->log_for_dirtybit = NULL;
>> >
>> > +#ifndef CONFIG_STUBDOM
>> >     /* Initialize backend core & drivers */
>> >     if (xen_be_init() != 0) {
>> >         fprintf(stderr, "%s: xen backend core setup failed\n", __FUNCTION__);
>> > @@ -930,7 +931,7 @@ int xen_hvm_init(void)
>> >     xen_be_register("console", &xen_console_ops);
>> >     xen_be_register("vkbd", &xen_kbdmouse_ops);
>> >     xen_be_register("qdisk", &xen_blkdev_ops);
>> > -
>> > +#endif
>> >     return 0;
>> >  }
>> >
>> >
>> >
>> >
>> > Yes. this implementation of stubdom is more closer to the old qemu's implementation. Which branch this patch works
>> on? The
>> > qemu-dm-v15 would not so lucky.
>>
>> That branch is old now, checkout this one:
>>
>> git://xenbits.xen.org/people/sstabellini/qemu-dm.git xen-stable-0.15
>>
>>
>>
>> Do you mean I migrate my work to this qemu branch?
>
> Yes, if it doesn't cost too much time for you to rebase your patches.


Hi Stefano,

     These days I updated my patches to support  xen-stable-0.15 qemu
and latest xen-unstable. The fbdev is still not work. However, I have
found three vram mapping areas in linux based stubdom. I think maybe
our problem is caused by the mismatch of these areas. I list them with
the order of initialization.

1.  vga_common_init(s,8M)
     This vram area is mapped by xc_map_foreign_bulk()
               ->linux_privcmd_map_foreign_bulk()
                        ->mmap(NULL, (unsigned long)num <<
XC_PAGE_SHIFT, prot, MAP_SHARED,fd, 0);
     The fd is descriptor of privcmd device. The
entry->vaddr_base=0xb6bdc000. Therefore, the mapping area is
0xb6bdc000 ~ 0xb73dc000, 8M.

     The block->offset allocated by qemu_get_ram_ptr() is ram_size.
The ram_size is the declared memory of HVM guest. In my case, the hvm
guest takes 384M memory. So block->offset=0x18000000. The guest
physical memory of vram is 0x18000000 ~ 0x18800000, 8M.

2. fbdev_init()
     This vram area is mapped by fbdev_init()

->mmap(NULL,fb_fix.smem_len+fb_mem_offset,PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
     The fd is descriptor of fb0 device. The mapping area is
0xb680a000 ~ 0xb6a0a000, 2M.

3. xen_add_to_physmap()
      This function is mapping the vram to 0xf0000000 ~ 0xf080000, 8M.

    Referring to minios-based stubdom, I think the stubdom will work
well if fbdev maps vram according to 0x18000000 ~ 0x18800000. I look
forward to your comments and suggestions.

    Thanks,


Jiageng Yu.

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/8/13 Jiageng Yu <yujiageng734@gmail.com>:
> 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
>> On Fri, 29 Jul 2011, Jiageng Yu wrote:
>>> 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>>> On Fri, 29 Jul 2011, Jiageng Yu wrote:
>>> > 2011/7/29 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>>> > On Fri, 29 Jul 2011, Jiageng Yu wrote:
>>> > > I have noticed the mistake. In fact, we shall stop the map_foreign_pages of xen_console and xenfb devices in
>>> qemu.
>>> > Because
>>> > > the front drivers in stubdom has already map the memories. This is my new patch. But it is not stable, I am
>>> > testing it.
>>> > >
>>> > > We use xc_handle to map foreign pages in xenfb and xen_console devices. If qemu running on stubdom, the
>>> xc_handle
>>> > is
>>> > > invalid.
>>> > >
>>> > >
>>> > > diff --git a/hw/xen_backend.c b/hw/xen_backend.c
>>> > > index cfb53c8..11c53fe 100644
>>> > > --- a/hw/xen_backend.c
>>> > > +++ b/hw/xen_backend.c
>>> > > @@ -47,6 +47,7 @@ XenXC xen_xc = XC_HANDLER_INITIAL_VALUE;
>>> > >  XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>>> > >  struct xs_handle *xenstore = NULL;
>>> > >  const char *xen_protocol;
>>> > > +XenXC xc_handle = XC_HANDLER_INITIAL_VALUE;
>>> > >
>>> > >  /* private */
>>> > >  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs = QTAILQ_HEAD_INITIALIZER(xendevs);
>>> > > @@ -655,6 +656,7 @@ static void xen_be_evtchn_event(void *opaque)
>>> > >
>>> > >  int xen_be_init(void)
>>> > >  {
>>> > > +#ifndef CONFIG_STUBDOM
>>> > >      xenstore = xs_daemon_open();
>>> > >      if (!xenstore) {
>>> > >          xen_be_printf(NULL, 0, "can't connect to xenstored\n");
>>> > > @@ -665,10 +667,16 @@ int xen_be_init(void)
>>> > >          goto err;
>>> > >      }
>>> > >
>>> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
>>> > > +        goto err;
>>> > > +    }
>>> > > +#endif
>>> > > +
>>> > >      if (xen_xc == XC_HANDLER_INITIAL_VALUE) {
>>> > >          /* Check if xen_init() have been called */
>>> > >          goto err;
>>> > >      }
>>> > > +
>>> > >      return 0;
>>> > >
>>> > >  err:
>>> > > diff --git a/hw/xen_backend.h b/hw/xen_backend.h
>>> > > index 9d36df3..bc5a157 100644
>>> > > --- a/hw/xen_backend.h
>>> > > +++ b/hw/xen_backend.h
>>> > > @@ -59,6 +59,9 @@ extern XenXC xen_xc;
>>> > >  extern struct xs_handle *xenstore;
>>> > >  extern const char *xen_protocol;
>>> > >
>>> > > +/* invalid in linux stubdom */
>>> > > +extern XenXC xc_handle;
>>> > > +
>>> > >  /* xenstore helper functions */
>>> > >  int xenstore_write_str(const char *base, const char *node, const char *val);
>>> > >  int xenstore_write_int(const char *base, const char *node, int ival);
>>> > > diff --git a/hw/xen_console.c b/hw/xen_console.c
>>> > > index c6c8163..66b6dd7 100644
>>> > > --- a/hw/xen_console.c
>>> > > +++ b/hw/xen_console.c
>>> > > @@ -213,7 +213,7 @@ static int con_connect(struct XenDevice *xendev)
>>> > >      if (xenstore_read_int(con->console, "limit", &limit) == 0)
>>> > >   con->buffer.max_capacity = limit;
>>> > >
>>> > > -    con->sring = xc_map_foreign_range(xen_xc, con->xendev.dom,
>>> > > +   con->sring = xc_map_foreign_range(xc_handle, con->xendev.dom,
>>> > >            XC_PAGE_SIZE,
>>> > >            PROT_READ|PROT_WRITE,
>>> > >            con->ring_ref);
>>> > > diff --git a/hw/xenfb.c b/hw/xenfb.c
>>> > > index 039076a..278fa60 100644
>>> > > --- a/hw/xenfb.c
>>> > > +++ b/hw/xenfb.c
>>> > > @@ -104,7 +104,7 @@ static int common_bind(struct common *c)
>>> > >      if (xenstore_read_fe_int(&c->xendev, "event-channel", &c->xendev.remote_port) == -1)
>>> > >   return -1;
>>> > >
>>> > > -    c->page = xc_map_foreign_range(xen_xc, c->xendev.dom,
>>> > > +   c->page = xc_map_foreign_range(xc_handle, c->xendev.dom,
>>> > >         XC_PAGE_SIZE,
>>> > >         PROT_READ | PROT_WRITE, mfn);
>>> > >      if (c->page == NULL)
>>> > > @@ -482,14 +482,14 @@ static int xenfb_map_fb(struct XenFB *xenfb)
>>> > >      fbmfns = qemu_mallocz(sizeof(unsigned long) * xenfb->fbpages);
>>> > >
>>> > >      xenfb_copy_mfns(mode, n_fbdirs, pgmfns, pd);
>>> > > -    map = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
>>> > > +   map = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>>> > >            PROT_READ, pgmfns, n_fbdirs);
>>> > >      if (map == NULL)
>>> > >   goto out;
>>> > >      xenfb_copy_mfns(mode, xenfb->fbpages, fbmfns, map);
>>> > >      munmap(map, n_fbdirs * XC_PAGE_SIZE);
>>> > >
>>> > > -    xenfb->pixels = xc_map_foreign_pages(xen_xc, xenfb->c.xendev.dom,
>>> > > +   xenfb->pixels = xc_map_foreign_pages(xc_handle, xenfb->c.xendev.dom,
>>> > >        PROT_READ | PROT_WRITE, fbmfns, xenfb->fbpages);
>>> > >      if (xenfb->pixels == NULL)
>>> > >   goto out;
>>> > > diff --git a/xen-all.c b/xen-all.c
>>> > > index b73fc43..04dfb51 100644
>>> > > --- a/xen-all.c
>>> > > +++ b/xen-all.c
>>> > > @@ -527,12 +534,22 @@ int xen_init(void)
>>> > >          return -1;
>>> > >      }
>>> > >
>>> > > +#ifdef CONFIG_STUBDOM
>>> > > +    return 0;
>>> > > +#endif
>>> > > +
>>> > > +    xc_handle = xen_xc_interface_open(0, 0, 0);
>>> > > +    if (xc_handle == XC_HANDLER_INITIAL_VALUE) {
>>> > > +        xen_be_printf(NULL, 0, "can't open xen interface\n");
>>> > > +        return -1;
>>> > > +    }
>>> > > +
>>> > >      return 0;
>>> > >  }
>>> > >
>>> > >
>>> >
>>> > I think that the backends shouldn't be initialized at all when running
>>> > in the stubdom, so I would do something like this instead:
>>> >
>>> > diff --git a/xen-all.c b/xen-all.c
>>> > index 167bed6..e3f630b 100644
>>> > --- a/xen-all.c
>>> > +++ b/xen-all.c
>>> > @@ -922,6 +922,7 @@ int xen_hvm_init(void)
>>> >     cpu_register_phys_memory_client(&state->client);
>>> >     state->log_for_dirtybit = NULL;
>>> >
>>> > +#ifndef CONFIG_STUBDOM
>>> >     /* Initialize backend core & drivers */
>>> >     if (xen_be_init() != 0) {
>>> >         fprintf(stderr, "%s: xen backend core setup failed\n", __FUNCTION__);
>>> > @@ -930,7 +931,7 @@ int xen_hvm_init(void)
>>> >     xen_be_register("console", &xen_console_ops);
>>> >     xen_be_register("vkbd", &xen_kbdmouse_ops);
>>> >     xen_be_register("qdisk", &xen_blkdev_ops);
>>> > -
>>> > +#endif
>>> >     return 0;
>>> >  }
>>> >
>>> >
>>> >
>>> >
>>> > Yes. this implementation of stubdom is more closer to the old qemu's implementation. Which branch this patch works
>>> on? The
>>> > qemu-dm-v15 would not so lucky.
>>>
>>> That branch is old now, checkout this one:
>>>
>>> git://xenbits.xen.org/people/sstabellini/qemu-dm.git xen-stable-0.15
>>>
>>>
>>>
>>> Do you mean I migrate my work to this qemu branch?
>>
>> Yes, if it doesn't cost too much time for you to rebase your patches.
>
>
> Hi Stefano,
>
>     These days I updated my patches to support  xen-stable-0.15 qemu
> and latest xen-unstable. The fbdev is still not work. However, I have
> found three vram mapping areas in linux based stubdom. I think maybe
> our problem is caused by the mismatch of these areas. I list them with
> the order of initialization.
>
> 1.  vga_common_init(s,8M)
>     This vram area is mapped by xc_map_foreign_bulk()
>               ->linux_privcmd_map_foreign_bulk()
>                        ->mmap(NULL, (unsigned long)num <<
> XC_PAGE_SHIFT, prot, MAP_SHARED,fd, 0);
>     The fd is descriptor of privcmd device. The
> entry->vaddr_base=0xb6bdc000. Therefore, the mapping area is
> 0xb6bdc000 ~ 0xb73dc000, 8M.
>
>     The block->offset allocated by qemu_get_ram_ptr() is ram_size.
> The ram_size is the declared memory of HVM guest. In my case, the hvm
> guest takes 384M memory. So block->offset=0x18000000. The guest
> physical memory of vram is 0x18000000 ~ 0x18800000, 8M.
>
> 2. fbdev_init()
>     This vram area is mapped by fbdev_init()
>
> ->mmap(NULL,fb_fix.smem_len+fb_mem_offset,PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
>     The fd is descriptor of fb0 device. The mapping area is
> 0xb680a000 ~ 0xb6a0a000, 2M.
>
> 3. xen_add_to_physmap()
>      This function is mapping the vram to 0xf0000000 ~ 0xf080000, 8M.
>
>    Referring to minios-based stubdom, I think the stubdom will work
> well if fbdev maps vram according to 0x18000000 ~ 0x18800000. I look
> forward to your comments and suggestions.
>
>    Thanks,
>
>
> Jiageng Yu.
>

Hi Stefano,

    In the linux-pv xenfbfront driver, the vram is allocated by:
    xenfb_probe()
        ->info->fb = vmalloc(fb_size);

    In the linux-stubdom, the memory areas: (info->fb,
info->fb+fb_size) in linux-pv kernel, (s->vram_offset,
s->vram_offset+VGA_RAM_SIZE) in vga_common_init function,
(s->vram_ptr, s->vram_ptr+VGA_RAM_SIZE) in stubdom. These memory areas
should be mapped into the same machine memory region.

    But the (info->fb, info->fb+fb_size) in linux-pv kernel is
allocated independently. I have two optional plans to slove the
problem.

1. Modify linux-pv kernel.
    This plan is more closer to the minios stubdom. First, I delay the
initial process of xenfbfront driver until qemu allocates s->vram_ptr.
Then, I set info->fb = s->vram_ptr.

2. Modify libxc.
    The s->vram_ptr is generated by mmap() function in
linux_privcmd_map_foreign_bulk(). Maybe I could replace the return
value of mmap() with the info->fb, which is obtained from xenstore.

    Can these plans solve the problem? Or there is the better one?

    Regards!

Jiageng Yu.

Re: Linux Stubdom Problem

[Stefano Stabellini]

On Mon, 15 Aug 2011, Jiageng Yu wrote:
> Hi Stefano,
> 
>     In the linux-pv xenfbfront driver, the vram is allocated by:
>     xenfb_probe()
>         ->info->fb = vmalloc(fb_size);
> 
>     In the linux-stubdom, the memory areas: (info->fb,
> info->fb+fb_size) in linux-pv kernel, (s->vram_offset,
> s->vram_offset+VGA_RAM_SIZE) in vga_common_init function,
> (s->vram_ptr, s->vram_ptr+VGA_RAM_SIZE) in stubdom. These memory areas
> should be mapped into the same machine memory region.
> 
>     But the (info->fb, info->fb+fb_size) in linux-pv kernel is
> allocated independently. I have two optional plans to slove the
> problem.
> 
> 1. Modify linux-pv kernel.
>     This plan is more closer to the minios stubdom. First, I delay the
> initial process of xenfbfront driver until qemu allocates s->vram_ptr.
> Then, I set info->fb = s->vram_ptr.
> 
> 2. Modify libxc.
>     The s->vram_ptr is generated by mmap() function in
> linux_privcmd_map_foreign_bulk(). Maybe I could replace the return
> value of mmap() with the info->fb, which is obtained from xenstore.

you would also need to modify qemu_ram_alloc to make sure the same pages
are remapped into the guest as videoram


>     Can these plans solve the problem? Or there is the better one?

I think it is best to keep the memory allocation in xenfbfront as it is
and find a way to map the pages in qemu. Isn't possible to use a mmap on
the fbdev device to obtain a mapping of the vmalloc'ed area into qemu?
Once we have mapped the pages into qemu address space it is just a
matter of using them as videoram.  Otherwise you would have timing
issues because qemu is probably going to start after xenfbfront has
already allocated the buffer and enstablished a connection with the
backend.

In any case why don't you try running a guest without a video card (the
qemu command line option for that is -vga none or -nographic) and see if
everything else works?

Re: Linux Stubdom Problem

[Jiageng Yu]

2011/8/19 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Mon, 15 Aug 2011, Jiageng Yu wrote:
>> Hi Stefano,
>>
>>     In the linux-pv xenfbfront driver, the vram is allocated by:
>>     xenfb_probe()
>>         ->info->fb = vmalloc(fb_size);
>>
>>     In the linux-stubdom, the memory areas: (info->fb,
>> info->fb+fb_size) in linux-pv kernel, (s->vram_offset,
>> s->vram_offset+VGA_RAM_SIZE) in vga_common_init function,
>> (s->vram_ptr, s->vram_ptr+VGA_RAM_SIZE) in stubdom. These memory areas
>> should be mapped into the same machine memory region.
>>
>>     But the (info->fb, info->fb+fb_size) in linux-pv kernel is
>> allocated independently. I have two optional plans to slove the
>> problem.
>>
>> 1. Modify linux-pv kernel.
>>     This plan is more closer to the minios stubdom. First, I delay the
>> initial process of xenfbfront driver until qemu allocates s->vram_ptr.
>> Then, I set info->fb = s->vram_ptr.
>>
>> 2. Modify libxc.
>>     The s->vram_ptr is generated by mmap() function in
>> linux_privcmd_map_foreign_bulk(). Maybe I could replace the return
>> value of mmap() with the info->fb, which is obtained from xenstore.
>
> you would also need to modify qemu_ram_alloc to make sure the same pages
> are remapped into the guest as videoram
>
>
>>     Can these plans solve the problem? Or there is the better one?
>
> I think it is best to keep the memory allocation in xenfbfront as it is
> and find a way to map the pages in qemu. Isn't possible to use a mmap on
> the fbdev device to obtain a mapping of the vmalloc'ed area into qemu?
> Once we have mapped the pages into qemu address space it is just a
> matter of using them as videoram.  Otherwise you would have timing
> issues because qemu is probably going to start after xenfbfront has
> already allocated the buffer and enstablished a connection with the
> backend.
>
> In any case why don't you try running a guest without a video card (the
> qemu command line option for that is -vga none or -nographic) and see if
> everything else works?
>

Hi Stefano,

     I am trying to fix the vram mapping issue. That is my new patch.
It is still needed to debug. Please check it for me and make sure I am
running on the right way.

    I define a new enmu type Stubdom_Vga_State which is used to notify
xen_remap_bucket whether to map the vram memory. In
fbdev_pv_display_prepare function, I map the xen_fbfront memory to
qemu (fb_mem) and directly incoke ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH,
&ioctlx) to map the vram of hvm guest.


diff --git a/fbdev.c b/fbdev.c
index a601b48..f7ff682 100644
--- a/fbdev.c
+++ b/fbdev.c
@@ -30,6 +30,12 @@
 #include "sysemu.h"
 #include "pflib.h"

+#include "hw/xen_backend.h"
+#include <xen/hvm/params.h>
+#include <sys/ioctl.h>
+#include "xenctrlosdep.h"
+#include <xen/privcmd.h>
+
 /* -------------------------------------------------------------------- */

 /* file handles */
@@ -541,29 +547,23 @@ static void fbdev_cleanup(void)
     }
 }

-static int fbdev_init(const char *device)
+static int fbdev_init(int prot, unsigned long size)
 {
     struct vt_stat vts;
	unsigned long page_mask;
     char ttyname[32];

     /* open framebuffer */
-    if (device == NULL) {
-       device = getenv("FRAMEBUFFER");
-    }
-    if (device == NULL) {
-       device = "/dev/fb0";
-    }
-    fb = open(device, O_RDWR);
+    fb = open("/dev/fb0", O_RDWR);
     if (fb == -1) {
-       fprintf(stderr, "open %s: %s\n", device, strerror(errno));
+        fprintf(stderr, "open /dev/fb0: %s\n", strerror(errno));
         return -1;
     }

     /* open virtual console */
     tty = 0;
     if (ioctl(tty, VT_GETSTATE, &vts) < 0) {
	     fprintf(stderr, "Not started from virtual terminal, trying to
open one.\n");

         snprintf(ttyname, sizeof(ttyname), "/dev/tty0");
         tty = open(ttyname, O_RDWR);
@@ -632,14 +632,14 @@ static int fbdev_init(const char *device)
        goto err;
     }

     page_mask = getpagesize()-1;
     fb_mem_offset = (unsigned long)(fb_fix.smem_start) & page_mask;
-    fb_mem = mmap(NULL,fb_fix.smem_len+fb_mem_offset,
-                 PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
+    fb_mem = mmap(NULL, size << XC_PAGE_SHIFT, prot, MAP_SHARED, fb, 0);
     if (fb_mem == MAP_FAILED) {
        perror("mmap");
        goto err;
     }
+
     /* move viewport to upper left corner */
     if (fb_var.xoffset != 0 || fb_var.yoffset != 0) {
        fb_var.xoffset = 0;
@@ -930,3 +928,71 @@ void fbdev_display_uninit(void)
     qemu_remove_exit_notifier(&exit_notifier);
     uninit_mouse();
 }
+
+int fbdev_pv_display_prepare(unsigned long domid, int prot, const
unsigned long *arr,
+								int *err, unsigned int num)
+{
+	xen_pfn_t *pfn;
+	privcmd_mmapbatch_t ioctlx;
+	int fd,i,rc;
+
+    if (fbdev_init(prot,num) != 0) {
+        exit(1);
+    }
+
+	pfn = malloc(num * sizeof(*pfn));
+	if(!pfn){
+		errno = -ENOMEM;
+		rc = -1;
+		return rc;
+	}
+	memcpy(pfn, arr, num*sizeof(*arr));
+
+	fd = open("/proc/xen/privcmd", O_RDWR);
+	if(fd == -1){
+		fprintf(stderr,"Could not obtain handle on privcmd device\n");
+		exit(1);
+	}
+
+	ioctlx.num = num;
+	ioctlx.dom = domid;
+	ioctlx.addr = (unsigned long)fb_mem;
+	ioctlx.arr = pfn;
+
+	rc = ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, &ioctlx);
+
+	for(i=0; i<num; i++)
+	{
+		switch(pfn[i] ^ arr[i])
+		{
+			case 0:
+				err[i] = rc != -ENOENT ? rc:0;
+				continue;
+			default:
+				err[i] = -EINVAL;
+				continue;
+		}
+		break;
+	}
+
+	free(pfn);
+
+	if (rc == -ENOENT && i == num)
+		rc=0;
+	else if(rc)
+	{
+		errno = -rc;
+		rc = -1;
+	}
+
+	if(rc < 0)
+	{
+		fprintf(stderr,"privcmd ioctl failed\n");
+		munmap(fb_mem, num << XC_PAGE_SHIFT);
+		return NULL;
+	}
+
+	close(fd);
+
+	return fb_mem;
+}
diff --git a/hw/vga.c b/hw/vga.c
index 0f54734..de7dd85 100644
--- a/hw/vga.c
+++ b/hw/vga.c
@@ -28,6 +28,7 @@
 #include "vga_int.h"
 #include "pixel_ops.h"
 #include "qemu-timer.h"
+#include "xen_backend.h"

 //#define DEBUG_VGA
 //#define DEBUG_VGA_MEM
@@ -2237,7 +2238,12 @@ void vga_common_init(VGACommonState *s, int vga_ram_size)
     s->is_vbe_vmstate = 0;
 #endif
     s->vram_offset = qemu_ram_alloc(NULL, "vga.vram", vga_ram_size);
+#ifdef CONFIG_STUBDOM
+	stubdom_vga_state = VGA_INIT_READY;
+#endif
     s->vram_ptr = qemu_get_ram_ptr(s->vram_offset);
     s->vram_size = vga_ram_size;
     s->get_bpp = vga_get_bpp;
     s->get_offsets = vga_get_offsets;
diff --git a/hw/xen_backend.c b/hw/xen_backend.c
index c506dfe..f4ecce4 100644
--- a/hw/xen_backend.c
+++ b/hw/xen_backend.c
@@ -48,6 +48,10 @@ XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
 struct xs_handle *xenstore = NULL;
 const char *xen_protocol;

+#ifdef CONFIG_STUBDOM
+enum Stubdom_Vga_State stubdom_vga_state=0;
+#endif
+
 /* private */
 static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
QTAILQ_HEAD_INITIALIZER(xendevs);
 static int debug = 0;
diff --git a/hw/xen_backend.h b/hw/xen_backend.h
index 3305630..36167d1 100644
--- a/hw/xen_backend.h
+++ b/hw/xen_backend.h
@@ -60,6 +60,16 @@ extern XenXC xen_xc;
 extern struct xs_handle *xenstore;
 extern const char *xen_protocol;

+#ifdef CONFIG_STUBDOM
+/* linux stubdom vga initialization*/
+enum Stubdom_Vga_State{
+	VGA_INIT_WAIT = 0,
+	VGA_INIT_READY,
+	VGA_INIT_COMPLETE
+};
+extern enum Stubdom_Vga_State stubdom_vga_state;
+#endif
+
 /* xenstore helper functions */
 int xenstore_write_str(const char *base, const char *node, const char *val);
 int xenstore_write_int(const char *base, const char *node, int ival);
diff --git a/xen-mapcache.c b/xen-mapcache.c
index 007136a..e615f98 100644
--- a/xen-mapcache.c
+++ b/xen-mapcache.c
@@ -20,6 +20,7 @@
 #include "xen-mapcache.h"
 #include "trace.h"

+#include "hw/xen.h"

 //#define MAPCACHE_DEBUG

@@ -144,8 +145,19 @@ static void xen_remap_bucket(MapCacheEntry *entry,
         pfns[i] = (address_index << (MCACHE_BUCKET_SHIFT-XC_PAGE_SHIFT)) + i;
     }

+#ifdef CONFIG_STUBDOM
+	if(stubdom_vga_state == VGA_INIT_READY){
+		fprintf(stderr,"xen_remap_bucket: start linux stubdom vga\n");
+		vaddr_base = fbdev_pv_display_prepare(xen_domid, PROT_READ|PROT_WRITE,
+										pfns, err, nb_pfn);
+		stubdom_vga_state = VGA_INIT_COMPLETE;
+	}else
+    	vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
+                                     	pfns, err, nb_pfn);
+#else
     vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
                                      pfns, err, nb_pfn);
+#endif
     if (vaddr_base == NULL) {
         perror("xc_map_foreign_bulk");
         exit(-1);


    Through I try to avoid modifing any code of linux-pv kernel, I
still have to commente out some code to make it run.

diff --git a/drivers/xen/xenfs/privcmd.c b/drivers/xen/xenfs/privcmd.c
index f80be7f..420f3b1 100644
--- a/drivers/xen/xenfs/privcmd.c
+++ b/drivers/xen/xenfs/privcmd.c
@@ -303,11 +309,10 @@ static long privcmd_ioctl_mmap_batch(void __user *udata)

 	vma = find_vma(mm, m.addr);
 	ret = -EINVAL;
+
 	if (!vma ||
-	    vma->vm_ops != &privcmd_vm_ops ||
 	    (m.addr != vma->vm_start) ||
-	    ((m.addr + (nr_pages << PAGE_SHIFT)) != vma->vm_end) ||
-	    !privcmd_enforce_singleshot_mapping(vma)) {
+	    ((m.addr + (nr_pages << PAGE_SHIFT)) != vma->vm_end)) {
 		up_write(&mm->mmap_sem);
 		goto out;
 	}


    The debug dump of qemu is:

(XEN) mm.c:945:d51 Error getting mfn 8ed68 (pfn 18000) from L1 entry
800000008ed68625 for l1e_owner=51, pg_owner=51
(XEN) mm.c:5044:d51 ptwr_emulate: fixing up invalid PAE PTE 800000008ed68625
(XEN) mm.c:945:d51 Error getting mfn 8ed68 (pfn 18000) from L1 entry
800000008ed68667 for l1e_owner=51, pg_owner=51
(XEN) mm.c:5049:d51 ptwr_emulate: could not get_page_from_l1e()
<1>BUG: unable to handle kernel <c>paging request<c> at c1f1b8f8
<1>IP: [<c011a7f0>] ptep_set_access_flags+0x40/0x80
*pdpt = 0000000001ee2001 <c>*pde = 0000000000008067 *pte = 8000000001f1b061
<0>Oops: 0003 [#1]
<0>last sysfs file: /sys/devices/virtual/vc/vcsa1/dev

Pid: 283, comm: qemu Not tainted (2.6.32.41 #6)
EIP: 0061:[<c011a7f0>] EFLAGS: 00010202 CPU: 0
EIP is at ptep_set_access_flags+0x40/0x80


     That seems the mapping is failed. (pfn 18000) is the first vram
page in hvm guest.


Jiageng Yu.

Re: Linux Stubdom Problem

[Stefano Stabellini]

On Mon, 22 Aug 2011, Jiageng Yu wrote:
> Hi Stefano,
> 
>      I am trying to fix the vram mapping issue. That is my new patch.
> It is still needed to debug. Please check it for me and make sure I am
> running on the right way.
> 
>     I define a new enmu type Stubdom_Vga_State which is used to notify
> xen_remap_bucket whether to map the vram memory. In
> fbdev_pv_display_prepare function, I map the xen_fbfront memory to
> qemu (fb_mem) and directly incoke ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH,
> &ioctlx) to map the vram of hvm guest.
> 

The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
mapping foreign mfns into the address space of the caller, while we need
to remap a set of pages already mapped in the address space of the
caller to some gmfns of a foreign domain. In other words we need the
same functionality but in the other direction.

First of all we need an hypercall to remap a given mfn to a guest gmfn
and currently there are no hypercalls that do that, so we need to add a
new one or extend an existing hypercall.
I suggest extending xc_domain_add_to_physmap with a new XENMAPSPACE,
called XENMAPSPACE_mfn that takes an mfn and maps it into a particular
gmfn.


>From the Xen point of view we need to add a new XENMAPSPACE_mfn case to
xen/arch/x86/mm.c:arch_memory_op, the basic implementation should be
something like the following (uncompiled, untested):

diff -r a79c1d5b946e xen/arch/x86/mm.c
--- a/xen/arch/x86/mm.c	Fri Aug 19 10:00:25 2011 +0100
+++ b/xen/arch/x86/mm.c	Mon Aug 22 17:51:41 2011 +0000
@@ -4618,6 +4618,16 @@ long arch_memory_op(int op, XEN_GUEST_HA
             page = mfn_to_page(mfn);
             break;
         }
+        case XENMAPSPACE_mfn:
+        {
+            if ( !IS_PRIV_FOR(current->domain, d) )
+                return -EINVAL;
+            mfn = xatp.idx;
+            page = mfn_to_page(mfn);
+            break;
+        }
         default:
             break;
         }
@@ -4648,10 +4658,12 @@ long arch_memory_op(int op, XEN_GUEST_HA
         }
 
         /* Unmap from old location, if any. */
-        gpfn = get_gpfn_from_mfn(mfn);
-        ASSERT( gpfn != SHARED_M2P_ENTRY );
-        if ( gpfn != INVALID_M2P_ENTRY )
-            guest_physmap_remove_page(d, gpfn, mfn, 0);
+        if ( xatp.space != XENMAPSPACE_mfn && d != current->domain ) {
+            gpfn = get_gpfn_from_mfn(mfn);
+            ASSERT( gpfn != SHARED_M2P_ENTRY );
+            if ( gpfn != INVALID_M2P_ENTRY )
+                guest_physmap_remove_page(d, gpfn, mfn, 0);
+        }
 
         /* Map at new location. */
         rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);


Unfortunately qemu doesn't know how to find the mfns corresponding to
the mmap'ed framebuffer and I would rather avoid writing a pagetable
walker in qemu.
Thus we need to use the linux kernel to do the virtual address to mfn
translation and issue the hypercall.
We can add a new privcmd ioctl IOCTL_PRIVCMD_FOREIGN_MAP: qemu calls this
ioctl with the mmap'ed framebuffer address, the size of the framebuffer
and the destination gmfns.
The implementation of the ioctl in the kernel would:

- find the list of mfns corresponding to the arguments, using
arbitrary_virt_to_machine;

- for each mfn, call the hypercall we extended with the appropriate
arguments, it would end up being
HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);

- there would be no "if (!xen_initial_domain())" check, because this
ioctl can be called by stubdoms.


So the call trace would be:

qemu: ioctl(fd, IOCTL_PRIVCMD_FOREIGN_MAP, &ioctlx);

|
v

linux: HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);

|
v

xen: guest_physmap_add_page


Has anybody any better ideas?


> diff --git a/fbdev.c b/fbdev.c
> index a601b48..f7ff682 100644
> --- a/fbdev.c
> +++ b/fbdev.c
> @@ -30,6 +30,12 @@
>  #include "sysemu.h"
>  #include "pflib.h"
> 
> +#include "hw/xen_backend.h"
> +#include <xen/hvm/params.h>
> +#include <sys/ioctl.h>
> +#include "xenctrlosdep.h"
> +#include <xen/privcmd.h>
> +
>  /* -------------------------------------------------------------------- */
> 
>  /* file handles */
> @@ -541,29 +547,23 @@ static void fbdev_cleanup(void)
>      }
>  }
> 
> -static int fbdev_init(const char *device)
> +static int fbdev_init(int prot, unsigned long size)
>  {
>      struct vt_stat vts;
>         unsigned long page_mask;
>      char ttyname[32];
> 
>      /* open framebuffer */
> -    if (device == NULL) {
> -       device = getenv("FRAMEBUFFER");
> -    }
> -    if (device == NULL) {
> -       device = "/dev/fb0";
> -    }
> -    fb = open(device, O_RDWR);
> +    fb = open("/dev/fb0", O_RDWR);
>      if (fb == -1) {
> -       fprintf(stderr, "open %s: %s\n", device, strerror(errno));
> +        fprintf(stderr, "open /dev/fb0: %s\n", strerror(errno));
>          return -1;
>      }
> 
>      /* open virtual console */
>      tty = 0;
>      if (ioctl(tty, VT_GETSTATE, &vts) < 0) {
>              fprintf(stderr, "Not started from virtual terminal, trying to
> open one.\n");
> 
>          snprintf(ttyname, sizeof(ttyname), "/dev/tty0");
>          tty = open(ttyname, O_RDWR);
> @@ -632,14 +632,14 @@ static int fbdev_init(const char *device)
>         goto err;
>      }
> 
>      page_mask = getpagesize()-1;
>      fb_mem_offset = (unsigned long)(fb_fix.smem_start) & page_mask;
> -    fb_mem = mmap(NULL,fb_fix.smem_len+fb_mem_offset,
> -                 PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
> +    fb_mem = mmap(NULL, size << XC_PAGE_SHIFT, prot, MAP_SHARED, fb, 0);
>      if (fb_mem == MAP_FAILED) {
>         perror("mmap");
>         goto err;
>      }
> +
>      /* move viewport to upper left corner */
>      if (fb_var.xoffset != 0 || fb_var.yoffset != 0) {
>         fb_var.xoffset = 0;
> @@ -930,3 +928,71 @@ void fbdev_display_uninit(void)
>      qemu_remove_exit_notifier(&exit_notifier);
>      uninit_mouse();
>  }

I would rather avoid modifing fbdev_init and add a new function to
xen-all.c that independently mmaps the xen_fbfront buffer with the right
size and maps it into the guest.


> +int fbdev_pv_display_prepare(unsigned long domid, int prot, const
> unsigned long *arr,
> +                                                               int *err, unsigned int num)
> +{
> +       xen_pfn_t *pfn;
> +       privcmd_mmapbatch_t ioctlx;
> +       int fd,i,rc;
> +
> +    if (fbdev_init(prot,num) != 0) {
> +        exit(1);
> +    }
> +
> +       pfn = malloc(num * sizeof(*pfn));
> +       if(!pfn){
> +               errno = -ENOMEM;
> +               rc = -1;
> +               return rc;
> +       }
> +       memcpy(pfn, arr, num*sizeof(*arr));
> +
> +       fd = open("/proc/xen/privcmd", O_RDWR);
> +       if(fd == -1){
> +               fprintf(stderr,"Could not obtain handle on privcmd device\n");
> +               exit(1);
> +       }
> +
> +       ioctlx.num = num;
> +       ioctlx.dom = domid;
> +       ioctlx.addr = (unsigned long)fb_mem;
> +       ioctlx.arr = pfn;
> +
> +       rc = ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, &ioctlx);
> +
> +       for(i=0; i<num; i++)
> +       {
> +               switch(pfn[i] ^ arr[i])
> +               {
> +                       case 0:
> +                               err[i] = rc != -ENOENT ? rc:0;
> +                               continue;
> +                       default:
> +                               err[i] = -EINVAL;
> +                               continue;
> +               }
> +               break;
> +       }
> +
> +       free(pfn);
> +
> +       if (rc == -ENOENT && i == num)
> +               rc=0;
> +       else if(rc)
> +       {
> +               errno = -rc;
> +               rc = -1;
> +       }
> +
> +       if(rc < 0)
> +       {
> +               fprintf(stderr,"privcmd ioctl failed\n");
> +               munmap(fb_mem, num << XC_PAGE_SHIFT);
> +               return NULL;
> +       }
> +
> +       close(fd);
> +
> +       return fb_mem;
> +}

We shouldn't use IOCTL_PRIVCMD_MMAPBATCH, we need a new ioctl, see above.


> diff --git a/hw/vga.c b/hw/vga.c
> index 0f54734..de7dd85 100644
> --- a/hw/vga.c
> +++ b/hw/vga.c
> @@ -28,6 +28,7 @@
>  #include "vga_int.h"
>  #include "pixel_ops.h"
>  #include "qemu-timer.h"
> +#include "xen_backend.h"
> 
>  //#define DEBUG_VGA
>  //#define DEBUG_VGA_MEM
> @@ -2237,7 +2238,12 @@ void vga_common_init(VGACommonState *s, int vga_ram_size)
>      s->is_vbe_vmstate = 0;
>  #endif
>      s->vram_offset = qemu_ram_alloc(NULL, "vga.vram", vga_ram_size);
> +#ifdef CONFIG_STUBDOM
> +       stubdom_vga_state = VGA_INIT_READY;
> +#endif
>      s->vram_ptr = qemu_get_ram_ptr(s->vram_offset);
>      s->vram_size = vga_ram_size;
>      s->get_bpp = vga_get_bpp;
>      s->get_offsets = vga_get_offsets;
> diff --git a/hw/xen_backend.c b/hw/xen_backend.c
> index c506dfe..f4ecce4 100644
> --- a/hw/xen_backend.c
> +++ b/hw/xen_backend.c
> @@ -48,6 +48,10 @@ XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>  struct xs_handle *xenstore = NULL;
>  const char *xen_protocol;
> 
> +#ifdef CONFIG_STUBDOM
> +enum Stubdom_Vga_State stubdom_vga_state=0;
> +#endif
> +
>  /* private */
>  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
> QTAILQ_HEAD_INITIALIZER(xendevs);
>  static int debug = 0;
> diff --git a/hw/xen_backend.h b/hw/xen_backend.h
> index 3305630..36167d1 100644
> --- a/hw/xen_backend.h
> +++ b/hw/xen_backend.h
> @@ -60,6 +60,16 @@ extern XenXC xen_xc;
>  extern struct xs_handle *xenstore;
>  extern const char *xen_protocol;
> 
> +#ifdef CONFIG_STUBDOM
> +/* linux stubdom vga initialization*/
> +enum Stubdom_Vga_State{
> +       VGA_INIT_WAIT = 0,
> +       VGA_INIT_READY,
> +       VGA_INIT_COMPLETE
> +};
> +extern enum Stubdom_Vga_State stubdom_vga_state;
> +#endif
> +
>  /* xenstore helper functions */
>  int xenstore_write_str(const char *base, const char *node, const char *val);
>  int xenstore_write_int(const char *base, const char *node, int ival);
> diff --git a/xen-mapcache.c b/xen-mapcache.c
> index 007136a..e615f98 100644
> --- a/xen-mapcache.c
> +++ b/xen-mapcache.c
> @@ -20,6 +20,7 @@
>  #include "xen-mapcache.h"
>  #include "trace.h"
> 
> +#include "hw/xen.h"
> 
>  //#define MAPCACHE_DEBUG
> 
> @@ -144,8 +145,19 @@ static void xen_remap_bucket(MapCacheEntry *entry,
>          pfns[i] = (address_index << (MCACHE_BUCKET_SHIFT-XC_PAGE_SHIFT)) + i;
>      }
> 
> +#ifdef CONFIG_STUBDOM
> +       if(stubdom_vga_state == VGA_INIT_READY){
> +               fprintf(stderr,"xen_remap_bucket: start linux stubdom vga\n");
> +               vaddr_base = fbdev_pv_display_prepare(xen_domid, PROT_READ|PROT_WRITE,
> +                                                                               pfns, err, nb_pfn);
> +               stubdom_vga_state = VGA_INIT_COMPLETE;
> +       }else
> +       vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
> +                                       pfns, err, nb_pfn);
> +#else
>      vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
>                                       pfns, err, nb_pfn);
> +#endif
>      if (vaddr_base == NULL) {
>          perror("xc_map_foreign_bulk");
>          exit(-1);
> 
> 

I don't like the stubdom_vga_init approach: in general it is a good idea
to avoid state machines unless necessary.
I would implement a new function called xen_vga_ram_map in xen-all.c
that mmaps the xen_fbfront buffer and uses the new ioctl to
map the buffer into the guest.
xen_vga_ram_map should be called instead of xen_ram_alloc by
qemu_ram_alloc_from_ptr if name == "vga.vram".


Another suggestion: before starting to write any new lines of code, I
would make sure that mmaping the /dev/fb device actually works correctly.
For debugging purposes you can try to modify fbdev_init and write
something to the framebuffer right after the mmap, to see if the new
pattern is displayed correctly on the screen.

Re: Re: Linux Stubdom Problem

[Keir Fraser]

On 22/08/2011 20:36, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
wrote:

> On Mon, 22 Aug 2011, Jiageng Yu wrote:
>> Hi Stefano,
>> 
>>      I am trying to fix the vram mapping issue. That is my new patch.
>> It is still needed to debug. Please check it for me and make sure I am
>> running on the right way.
>> 
>>     I define a new enmu type Stubdom_Vga_State which is used to notify
>> xen_remap_bucket whether to map the vram memory. In
>> fbdev_pv_display_prepare function, I map the xen_fbfront memory to
>> qemu (fb_mem) and directly incoke ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH,
>> &ioctlx) to map the vram of hvm guest.
>> 
> 
> The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
> mapping foreign mfns into the address space of the caller, while we need
> to remap a set of pages already mapped in the address space of the
> caller to some gmfns of a foreign domain. In other words we need the
> same functionality but in the other direction.
> 
> First of all we need an hypercall to remap a given mfn to a guest gmfn
> and currently there are no hypercalls that do that, so we need to add a
> new one or extend an existing hypercall.
> I suggest extending xc_domain_add_to_physmap with a new XENMAPSPACE,
> called XENMAPSPACE_mfn that takes an mfn and maps it into a particular
> gmfn.

I suggest XENMAPSPACE_caller_gmfn. I'm slightly worried about the reference
counting implications of mapping these foreign pages into an HVM guest. It
might need checking with Tim Deegan.
 
> 
>> From the Xen point of view we need to add a new XENMAPSPACE_mfn case to
> xen/arch/x86/mm.c:arch_memory_op, the basic implementation should be
> something like the following (uncompiled, untested):
> 
> diff -r a79c1d5b946e xen/arch/x86/mm.c
> --- a/xen/arch/x86/mm.c Fri Aug 19 10:00:25 2011 +0100
> +++ b/xen/arch/x86/mm.c Mon Aug 22 17:51:41 2011 +0000
> @@ -4618,6 +4618,16 @@ long arch_memory_op(int op, XEN_GUEST_HA
>              page = mfn_to_page(mfn);
>              break;
>          }
> +        case XENMAPSPACE_mfn:
> +        {
> +            if ( !IS_PRIV_FOR(current->domain, d) )
> +                return -EINVAL;
> +            mfn = xatp.idx;
> +            page = mfn_to_page(mfn);

Also need something like get_page(page, current->domain).

> +            break;
> +        }
>          default:
>              break;
>          }
> @@ -4648,10 +4658,12 @@ long arch_memory_op(int op, XEN_GUEST_HA
>          }
>  
>          /* Unmap from old location, if any. */
> -        gpfn = get_gpfn_from_mfn(mfn);
> -        ASSERT( gpfn != SHARED_M2P_ENTRY );
> -        if ( gpfn != INVALID_M2P_ENTRY )
> -            guest_physmap_remove_page(d, gpfn, mfn, 0);
> +        if ( xatp.space != XENMAPSPACE_mfn && d != current->domain ) {
> +            gpfn = get_gpfn_from_mfn(mfn);
> +            ASSERT( gpfn != SHARED_M2P_ENTRY );
> +            if ( gpfn != INVALID_M2P_ENTRY )
> +                guest_physmap_remove_page(d, gpfn, mfn, 0);
> +        }
>  
>          /* Map at new location. */
>          rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);
> 
> 
> Unfortunately qemu doesn't know how to find the mfns corresponding to
> the mmap'ed framebuffer and I would rather avoid writing a pagetable
> walker in qemu.
> Thus we need to use the linux kernel to do the virtual address to mfn
> translation and issue the hypercall.
> We can add a new privcmd ioctl IOCTL_PRIVCMD_FOREIGN_MAP: qemu calls this
> ioctl with the mmap'ed framebuffer address, the size of the framebuffer
> and the destination gmfns.
> The implementation of the ioctl in the kernel would:
> 
> - find the list of mfns corresponding to the arguments, using
> arbitrary_virt_to_machine;
> 
> - for each mfn, call the hypercall we extended with the appropriate
> arguments, it would end up being
> HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
> 
> - there would be no "if (!xen_initial_domain())" check, because this
> ioctl can be called by stubdoms.
> 
> 
> So the call trace would be:
> 
> qemu: ioctl(fd, IOCTL_PRIVCMD_FOREIGN_MAP, &ioctlx);
> 
> |
> v
> 
> linux: HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
> 
> |
> v
> 
> xen: guest_physmap_add_page
> 
> 
> Has anybody any better ideas?
> 
> 
>> diff --git a/fbdev.c b/fbdev.c
>> index a601b48..f7ff682 100644
>> --- a/fbdev.c
>> +++ b/fbdev.c
>> @@ -30,6 +30,12 @@
>>  #include "sysemu.h"
>>  #include "pflib.h"
>> 
>> +#include "hw/xen_backend.h"
>> +#include <xen/hvm/params.h>
>> +#include <sys/ioctl.h>
>> +#include "xenctrlosdep.h"
>> +#include <xen/privcmd.h>
>> +
>>  /* -------------------------------------------------------------------- */
>> 
>>  /* file handles */
>> @@ -541,29 +547,23 @@ static void fbdev_cleanup(void)
>>      }
>>  }
>> 
>> -static int fbdev_init(const char *device)
>> +static int fbdev_init(int prot, unsigned long size)
>>  {
>>      struct vt_stat vts;
>>         unsigned long page_mask;
>>      char ttyname[32];
>> 
>>      /* open framebuffer */
>> -    if (device == NULL) {
>> -       device = getenv("FRAMEBUFFER");
>> -    }
>> -    if (device == NULL) {
>> -       device = "/dev/fb0";
>> -    }
>> -    fb = open(device, O_RDWR);
>> +    fb = open("/dev/fb0", O_RDWR);
>>      if (fb == -1) {
>> -       fprintf(stderr, "open %s: %s\n", device, strerror(errno));
>> +        fprintf(stderr, "open /dev/fb0: %s\n", strerror(errno));
>>          return -1;
>>      }
>> 
>>      /* open virtual console */
>>      tty = 0;
>>      if (ioctl(tty, VT_GETSTATE, &vts) < 0) {
>>              fprintf(stderr, "Not started from virtual terminal, trying to
>> open one.\n");
>> 
>>          snprintf(ttyname, sizeof(ttyname), "/dev/tty0");
>>          tty = open(ttyname, O_RDWR);
>> @@ -632,14 +632,14 @@ static int fbdev_init(const char *device)
>>         goto err;
>>      }
>> 
>>      page_mask = getpagesize()-1;
>>      fb_mem_offset = (unsigned long)(fb_fix.smem_start) & page_mask;
>> -    fb_mem = mmap(NULL,fb_fix.smem_len+fb_mem_offset,
>> -                 PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
>> +    fb_mem = mmap(NULL, size << XC_PAGE_SHIFT, prot, MAP_SHARED, fb, 0);
>>      if (fb_mem == MAP_FAILED) {
>>         perror("mmap");
>>         goto err;
>>      }
>> +
>>      /* move viewport to upper left corner */
>>      if (fb_var.xoffset != 0 || fb_var.yoffset != 0) {
>>         fb_var.xoffset = 0;
>> @@ -930,3 +928,71 @@ void fbdev_display_uninit(void)
>>      qemu_remove_exit_notifier(&exit_notifier);
>>      uninit_mouse();
>>  }
> 
> I would rather avoid modifing fbdev_init and add a new function to
> xen-all.c that independently mmaps the xen_fbfront buffer with the right
> size and maps it into the guest.
> 
> 
>> +int fbdev_pv_display_prepare(unsigned long domid, int prot, const
>> unsigned long *arr,
>> +                                                               int *err,
>> unsigned int num)
>> +{
>> +       xen_pfn_t *pfn;
>> +       privcmd_mmapbatch_t ioctlx;
>> +       int fd,i,rc;
>> +
>> +    if (fbdev_init(prot,num) != 0) {
>> +        exit(1);
>> +    }
>> +
>> +       pfn = malloc(num * sizeof(*pfn));
>> +       if(!pfn){
>> +               errno = -ENOMEM;
>> +               rc = -1;
>> +               return rc;
>> +       }
>> +       memcpy(pfn, arr, num*sizeof(*arr));
>> +
>> +       fd = open("/proc/xen/privcmd", O_RDWR);
>> +       if(fd == -1){
>> +               fprintf(stderr,"Could not obtain handle on privcmd
>> device\n");
>> +               exit(1);
>> +       }
>> +
>> +       ioctlx.num = num;
>> +       ioctlx.dom = domid;
>> +       ioctlx.addr = (unsigned long)fb_mem;
>> +       ioctlx.arr = pfn;
>> +
>> +       rc = ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, &ioctlx);
>> +
>> +       for(i=0; i<num; i++)
>> +       {
>> +               switch(pfn[i] ^ arr[i])
>> +               {
>> +                       case 0:
>> +                               err[i] = rc != -ENOENT ? rc:0;
>> +                               continue;
>> +                       default:
>> +                               err[i] = -EINVAL;
>> +                               continue;
>> +               }
>> +               break;
>> +       }
>> +
>> +       free(pfn);
>> +
>> +       if (rc == -ENOENT && i == num)
>> +               rc=0;
>> +       else if(rc)
>> +       {
>> +               errno = -rc;
>> +               rc = -1;
>> +       }
>> +
>> +       if(rc < 0)
>> +       {
>> +               fprintf(stderr,"privcmd ioctl failed\n");
>> +               munmap(fb_mem, num << XC_PAGE_SHIFT);
>> +               return NULL;
>> +       }
>> +
>> +       close(fd);
>> +
>> +       return fb_mem;
>> +}
> 
> We shouldn't use IOCTL_PRIVCMD_MMAPBATCH, we need a new ioctl, see above.
> 
> 
>> diff --git a/hw/vga.c b/hw/vga.c
>> index 0f54734..de7dd85 100644
>> --- a/hw/vga.c
>> +++ b/hw/vga.c
>> @@ -28,6 +28,7 @@
>>  #include "vga_int.h"
>>  #include "pixel_ops.h"
>>  #include "qemu-timer.h"
>> +#include "xen_backend.h"
>> 
>>  //#define DEBUG_VGA
>>  //#define DEBUG_VGA_MEM
>> @@ -2237,7 +2238,12 @@ void vga_common_init(VGACommonState *s, int
>> vga_ram_size)
>>      s->is_vbe_vmstate = 0;
>>  #endif
>>      s->vram_offset = qemu_ram_alloc(NULL, "vga.vram", vga_ram_size);
>> +#ifdef CONFIG_STUBDOM
>> +       stubdom_vga_state = VGA_INIT_READY;
>> +#endif
>>      s->vram_ptr = qemu_get_ram_ptr(s->vram_offset);
>>      s->vram_size = vga_ram_size;
>>      s->get_bpp = vga_get_bpp;
>>      s->get_offsets = vga_get_offsets;
>> diff --git a/hw/xen_backend.c b/hw/xen_backend.c
>> index c506dfe..f4ecce4 100644
>> --- a/hw/xen_backend.c
>> +++ b/hw/xen_backend.c
>> @@ -48,6 +48,10 @@ XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>>  struct xs_handle *xenstore = NULL;
>>  const char *xen_protocol;
>> 
>> +#ifdef CONFIG_STUBDOM
>> +enum Stubdom_Vga_State stubdom_vga_state=0;
>> +#endif
>> +
>>  /* private */
>>  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
>> QTAILQ_HEAD_INITIALIZER(xendevs);
>>  static int debug = 0;
>> diff --git a/hw/xen_backend.h b/hw/xen_backend.h
>> index 3305630..36167d1 100644
>> --- a/hw/xen_backend.h
>> +++ b/hw/xen_backend.h
>> @@ -60,6 +60,16 @@ extern XenXC xen_xc;
>>  extern struct xs_handle *xenstore;
>>  extern const char *xen_protocol;
>> 
>> +#ifdef CONFIG_STUBDOM
>> +/* linux stubdom vga initialization*/
>> +enum Stubdom_Vga_State{
>> +       VGA_INIT_WAIT = 0,
>> +       VGA_INIT_READY,
>> +       VGA_INIT_COMPLETE
>> +};
>> +extern enum Stubdom_Vga_State stubdom_vga_state;
>> +#endif
>> +
>>  /* xenstore helper functions */
>>  int xenstore_write_str(const char *base, const char *node, const char *val);
>>  int xenstore_write_int(const char *base, const char *node, int ival);
>> diff --git a/xen-mapcache.c b/xen-mapcache.c
>> index 007136a..e615f98 100644
>> --- a/xen-mapcache.c
>> +++ b/xen-mapcache.c
>> @@ -20,6 +20,7 @@
>>  #include "xen-mapcache.h"
>>  #include "trace.h"
>> 
>> +#include "hw/xen.h"
>> 
>>  //#define MAPCACHE_DEBUG
>> 
>> @@ -144,8 +145,19 @@ static void xen_remap_bucket(MapCacheEntry *entry,
>>          pfns[i] = (address_index << (MCACHE_BUCKET_SHIFT-XC_PAGE_SHIFT)) +
>> i;
>>      }
>> 
>> +#ifdef CONFIG_STUBDOM
>> +       if(stubdom_vga_state == VGA_INIT_READY){
>> +               fprintf(stderr,"xen_remap_bucket: start linux stubdom
>> vga\n");
>> +               vaddr_base = fbdev_pv_display_prepare(xen_domid,
>> PROT_READ|PROT_WRITE,
>> +               
>> pfns, err, nb_pfn);
>> +               stubdom_vga_state = VGA_INIT_COMPLETE;
>> +       }else
>> +       vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid,
>> PROT_READ|PROT_WRITE,
>> +                                       pfns, err, nb_pfn);
>> +#else
>>      vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid,
>> PROT_READ|PROT_WRITE,
>>                                       pfns, err, nb_pfn);
>> +#endif
>>      if (vaddr_base == NULL) {
>>          perror("xc_map_foreign_bulk");
>>          exit(-1);
>> 
>> 
> 
> I don't like the stubdom_vga_init approach: in general it is a good idea
> to avoid state machines unless necessary.
> I would implement a new function called xen_vga_ram_map in xen-all.c
> that mmaps the xen_fbfront buffer and uses the new ioctl to
> map the buffer into the guest.
> xen_vga_ram_map should be called instead of xen_ram_alloc by
> qemu_ram_alloc_from_ptr if name == "vga.vram".
> 
> 
> Another suggestion: before starting to write any new lines of code, I
> would make sure that mmaping the /dev/fb device actually works correctly.
> For debugging purposes you can try to modify fbdev_init and write
> something to the framebuffer right after the mmap, to see if the new
> pattern is displayed correctly on the screen.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1: Type: text/plain, Size: 14277 bytes --]

2011/8/23 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Mon, 22 Aug 2011, Jiageng Yu wrote:
>> Hi Stefano,
>>
>>      I am trying to fix the vram mapping issue. That is my new patch.
>> It is still needed to debug. Please check it for me and make sure I am
>> running on the right way.
>>
>>     I define a new enmu type Stubdom_Vga_State which is used to notify
>> xen_remap_bucket whether to map the vram memory. In
>> fbdev_pv_display_prepare function, I map the xen_fbfront memory to
>> qemu (fb_mem) and directly incoke ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH,
>> &ioctlx) to map the vram of hvm guest.
>>
>
> The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
> mapping foreign mfns into the address space of the caller, while we need
> to remap a set of pages already mapped in the address space of the
> caller to some gmfns of a foreign domain. In other words we need the
> same functionality but in the other direction.
>
> First of all we need an hypercall to remap a given mfn to a guest gmfn
> and currently there are no hypercalls that do that, so we need to add a
> new one or extend an existing hypercall.
> I suggest extending xc_domain_add_to_physmap with a new XENMAPSPACE,
> called XENMAPSPACE_mfn that takes an mfn and maps it into a particular
> gmfn.
>
>
> From the Xen point of view we need to add a new XENMAPSPACE_mfn case to
> xen/arch/x86/mm.c:arch_memory_op, the basic implementation should be
> something like the following (uncompiled, untested):
>
> diff -r a79c1d5b946e xen/arch/x86/mm.c
> --- a/xen/arch/x86/mm.c Fri Aug 19 10:00:25 2011 +0100
> +++ b/xen/arch/x86/mm.c Mon Aug 22 17:51:41 2011 +0000
> @@ -4618,6 +4618,16 @@ long arch_memory_op(int op, XEN_GUEST_HA
>             page = mfn_to_page(mfn);
>             break;
>         }
> +        case XENMAPSPACE_mfn:
> +        {
> +            if ( !IS_PRIV_FOR(current->domain, d) )
> +                return -EINVAL;
> +            mfn = xatp.idx;
> +            page = mfn_to_page(mfn);
> +            break;
> +        }
>         default:
>             break;
>         }
> @@ -4648,10 +4658,12 @@ long arch_memory_op(int op, XEN_GUEST_HA
>         }
>
>         /* Unmap from old location, if any. */
> -        gpfn = get_gpfn_from_mfn(mfn);
> -        ASSERT( gpfn != SHARED_M2P_ENTRY );
> -        if ( gpfn != INVALID_M2P_ENTRY )
> -            guest_physmap_remove_page(d, gpfn, mfn, 0);
> +        if ( xatp.space != XENMAPSPACE_mfn && d != current->domain ) {
> +            gpfn = get_gpfn_from_mfn(mfn);
> +            ASSERT( gpfn != SHARED_M2P_ENTRY );
> +            if ( gpfn != INVALID_M2P_ENTRY )
> +                guest_physmap_remove_page(d, gpfn, mfn, 0);
> +        }
>
>         /* Map at new location. */
>         rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);
>
>
> Unfortunately qemu doesn't know how to find the mfns corresponding to
> the mmap'ed framebuffer and I would rather avoid writing a pagetable
> walker in qemu.
> Thus we need to use the linux kernel to do the virtual address to mfn
> translation and issue the hypercall.
> We can add a new privcmd ioctl IOCTL_PRIVCMD_FOREIGN_MAP: qemu calls this
> ioctl with the mmap'ed framebuffer address, the size of the framebuffer
> and the destination gmfns.
> The implementation of the ioctl in the kernel would:
>
> - find the list of mfns corresponding to the arguments, using
> arbitrary_virt_to_machine;
>
> - for each mfn, call the hypercall we extended with the appropriate
> arguments, it would end up being
> HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
>
> - there would be no "if (!xen_initial_domain())" check, because this
> ioctl can be called by stubdoms.
>
>
> So the call trace would be:
>
> qemu: ioctl(fd, IOCTL_PRIVCMD_FOREIGN_MAP, &ioctlx);
>
> |
> v
>
> linux: HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
>
> |
> v
>
> xen: guest_physmap_add_page


Yes. I am already working on it.

>
>
> Has anybody any better ideas?
>
>
>> diff --git a/fbdev.c b/fbdev.c
>> index a601b48..f7ff682 100644
>> --- a/fbdev.c
>> +++ b/fbdev.c
>> @@ -30,6 +30,12 @@
>>  #include "sysemu.h"
>>  #include "pflib.h"
>>
>> +#include "hw/xen_backend.h"
>> +#include <xen/hvm/params.h>
>> +#include <sys/ioctl.h>
>> +#include "xenctrlosdep.h"
>> +#include <xen/privcmd.h>
>> +
>>  /* -------------------------------------------------------------------- */
>>
>>  /* file handles */
>> @@ -541,29 +547,23 @@ static void fbdev_cleanup(void)
>>      }
>>  }
>>
>> -static int fbdev_init(const char *device)
>> +static int fbdev_init(int prot, unsigned long size)
>>  {
>>      struct vt_stat vts;
>>         unsigned long page_mask;
>>      char ttyname[32];
>>
>>      /* open framebuffer */
>> -    if (device == NULL) {
>> -       device = getenv("FRAMEBUFFER");
>> -    }
>> -    if (device == NULL) {
>> -       device = "/dev/fb0";
>> -    }
>> -    fb = open(device, O_RDWR);
>> +    fb = open("/dev/fb0", O_RDWR);
>>      if (fb == -1) {
>> -       fprintf(stderr, "open %s: %s\n", device, strerror(errno));
>> +        fprintf(stderr, "open /dev/fb0: %s\n", strerror(errno));
>>          return -1;
>>      }
>>
>>      /* open virtual console */
>>      tty = 0;
>>      if (ioctl(tty, VT_GETSTATE, &vts) < 0) {
>>              fprintf(stderr, "Not started from virtual terminal, trying to
>> open one.\n");
>>
>>          snprintf(ttyname, sizeof(ttyname), "/dev/tty0");
>>          tty = open(ttyname, O_RDWR);
>> @@ -632,14 +632,14 @@ static int fbdev_init(const char *device)
>>         goto err;
>>      }
>>
>>      page_mask = getpagesize()-1;
>>      fb_mem_offset = (unsigned long)(fb_fix.smem_start) & page_mask;
>> -    fb_mem = mmap(NULL,fb_fix.smem_len+fb_mem_offset,
>> -                 PROT_READ|PROT_WRITE,MAP_SHARED,fb,0);
>> +    fb_mem = mmap(NULL, size << XC_PAGE_SHIFT, prot, MAP_SHARED, fb, 0);
>>      if (fb_mem == MAP_FAILED) {
>>         perror("mmap");
>>         goto err;
>>      }
>> +
>>      /* move viewport to upper left corner */
>>      if (fb_var.xoffset != 0 || fb_var.yoffset != 0) {
>>         fb_var.xoffset = 0;
>> @@ -930,3 +928,71 @@ void fbdev_display_uninit(void)
>>      qemu_remove_exit_notifier(&exit_notifier);
>>      uninit_mouse();
>>  }
>
> I would rather avoid modifing fbdev_init and add a new function to
> xen-all.c that independently mmaps the xen_fbfront buffer with the right
> size and maps it into the guest.
>
>
>> +int fbdev_pv_display_prepare(unsigned long domid, int prot, const
>> unsigned long *arr,
>> +                                                               int *err, unsigned int num)
>> +{
>> +       xen_pfn_t *pfn;
>> +       privcmd_mmapbatch_t ioctlx;
>> +       int fd,i,rc;
>> +
>> +    if (fbdev_init(prot,num) != 0) {
>> +        exit(1);
>> +    }
>> +
>> +       pfn = malloc(num * sizeof(*pfn));
>> +       if(!pfn){
>> +               errno = -ENOMEM;
>> +               rc = -1;
>> +               return rc;
>> +       }
>> +       memcpy(pfn, arr, num*sizeof(*arr));
>> +
>> +       fd = open("/proc/xen/privcmd", O_RDWR);
>> +       if(fd == -1){
>> +               fprintf(stderr,"Could not obtain handle on privcmd device\n");
>> +               exit(1);
>> +       }
>> +
>> +       ioctlx.num = num;
>> +       ioctlx.dom = domid;
>> +       ioctlx.addr = (unsigned long)fb_mem;
>> +       ioctlx.arr = pfn;
>> +
>> +       rc = ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, &ioctlx);
>> +
>> +       for(i=0; i<num; i++)
>> +       {
>> +               switch(pfn[i] ^ arr[i])
>> +               {
>> +                       case 0:
>> +                               err[i] = rc != -ENOENT ? rc:0;
>> +                               continue;
>> +                       default:
>> +                               err[i] = -EINVAL;
>> +                               continue;
>> +               }
>> +               break;
>> +       }
>> +
>> +       free(pfn);
>> +
>> +       if (rc == -ENOENT && i == num)
>> +               rc=0;
>> +       else if(rc)
>> +       {
>> +               errno = -rc;
>> +               rc = -1;
>> +       }
>> +
>> +       if(rc < 0)
>> +       {
>> +               fprintf(stderr,"privcmd ioctl failed\n");
>> +               munmap(fb_mem, num << XC_PAGE_SHIFT);
>> +               return NULL;
>> +       }
>> +
>> +       close(fd);
>> +
>> +       return fb_mem;
>> +}
>
> We shouldn't use IOCTL_PRIVCMD_MMAPBATCH, we need a new ioctl, see above.
>
>
>> diff --git a/hw/vga.c b/hw/vga.c
>> index 0f54734..de7dd85 100644
>> --- a/hw/vga.c
>> +++ b/hw/vga.c
>> @@ -28,6 +28,7 @@
>>  #include "vga_int.h"
>>  #include "pixel_ops.h"
>>  #include "qemu-timer.h"
>> +#include "xen_backend.h"
>>
>>  //#define DEBUG_VGA
>>  //#define DEBUG_VGA_MEM
>> @@ -2237,7 +2238,12 @@ void vga_common_init(VGACommonState *s, int vga_ram_size)
>>      s->is_vbe_vmstate = 0;
>>  #endif
>>      s->vram_offset = qemu_ram_alloc(NULL, "vga.vram", vga_ram_size);
>> +#ifdef CONFIG_STUBDOM
>> +       stubdom_vga_state = VGA_INIT_READY;
>> +#endif
>>      s->vram_ptr = qemu_get_ram_ptr(s->vram_offset);
>>      s->vram_size = vga_ram_size;
>>      s->get_bpp = vga_get_bpp;
>>      s->get_offsets = vga_get_offsets;
>> diff --git a/hw/xen_backend.c b/hw/xen_backend.c
>> index c506dfe..f4ecce4 100644
>> --- a/hw/xen_backend.c
>> +++ b/hw/xen_backend.c
>> @@ -48,6 +48,10 @@ XenGnttab xen_xcg = XC_HANDLER_INITIAL_VALUE;
>>  struct xs_handle *xenstore = NULL;
>>  const char *xen_protocol;
>>
>> +#ifdef CONFIG_STUBDOM
>> +enum Stubdom_Vga_State stubdom_vga_state=0;
>> +#endif
>> +
>>  /* private */
>>  static QTAILQ_HEAD(XenDeviceHead, XenDevice) xendevs =
>> QTAILQ_HEAD_INITIALIZER(xendevs);
>>  static int debug = 0;
>> diff --git a/hw/xen_backend.h b/hw/xen_backend.h
>> index 3305630..36167d1 100644
>> --- a/hw/xen_backend.h
>> +++ b/hw/xen_backend.h
>> @@ -60,6 +60,16 @@ extern XenXC xen_xc;
>>  extern struct xs_handle *xenstore;
>>  extern const char *xen_protocol;
>>
>> +#ifdef CONFIG_STUBDOM
>> +/* linux stubdom vga initialization*/
>> +enum Stubdom_Vga_State{
>> +       VGA_INIT_WAIT = 0,
>> +       VGA_INIT_READY,
>> +       VGA_INIT_COMPLETE
>> +};
>> +extern enum Stubdom_Vga_State stubdom_vga_state;
>> +#endif
>> +
>>  /* xenstore helper functions */
>>  int xenstore_write_str(const char *base, const char *node, const char *val);
>>  int xenstore_write_int(const char *base, const char *node, int ival);
>> diff --git a/xen-mapcache.c b/xen-mapcache.c
>> index 007136a..e615f98 100644
>> --- a/xen-mapcache.c
>> +++ b/xen-mapcache.c
>> @@ -20,6 +20,7 @@
>>  #include "xen-mapcache.h"
>>  #include "trace.h"
>>
>> +#include "hw/xen.h"
>>
>>  //#define MAPCACHE_DEBUG
>>
>> @@ -144,8 +145,19 @@ static void xen_remap_bucket(MapCacheEntry *entry,
>>          pfns[i] = (address_index << (MCACHE_BUCKET_SHIFT-XC_PAGE_SHIFT)) + i;
>>      }
>>
>> +#ifdef CONFIG_STUBDOM
>> +       if(stubdom_vga_state == VGA_INIT_READY){
>> +               fprintf(stderr,"xen_remap_bucket: start linux stubdom vga\n");
>> +               vaddr_base = fbdev_pv_display_prepare(xen_domid, PROT_READ|PROT_WRITE,
>> +                                                                               pfns, err, nb_pfn);
>> +               stubdom_vga_state = VGA_INIT_COMPLETE;
>> +       }else
>> +       vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
>> +                                       pfns, err, nb_pfn);
>> +#else
>>      vaddr_base = xc_map_foreign_bulk(xen_xc, xen_domid, PROT_READ|PROT_WRITE,
>>                                       pfns, err, nb_pfn);
>> +#endif
>>      if (vaddr_base == NULL) {
>>          perror("xc_map_foreign_bulk");
>>          exit(-1);
>>
>>
>
> I don't like the stubdom_vga_init approach: in general it is a good idea
> to avoid state machines unless necessary.
> I would implement a new function called xen_vga_ram_map in xen-all.c
> that mmaps the xen_fbfront buffer and uses the new ioctl to
> map the buffer into the guest.
> xen_vga_ram_map should be called instead of xen_ram_alloc by
> qemu_ram_alloc_from_ptr if name == "vga.vram".
>

I have question here. I think xen_vga_ram_map() is used to instead of
xc_map_foreign_bulk() which maps hvm guest's vram.

vga_common_init
    -->qemu_get_ram_ptr
             -->xen_map_cache
                     -->xen_remap_bucket
                               -->xc_map_foreign_bulk

The xen_ram_alloc() calls xc_domain_populate_physmap_exact() to
increase the physical memory of hvm guest. And the increased physical
memory becomes hvm guest's vram.

For the xen_remap_bucket(), there is no "vga.vram" input parameter. So
we need the notification mechanism to invoke xen_vga_ram_map().


>
> Another suggestion: before starting to write any new lines of code, I
> would make sure that mmaping the /dev/fb device actually works correctly.
> For debugging purposes you can try to modify fbdev_init and write
> something to the framebuffer right after the mmap, to see if the new
> pattern is displayed correctly on the screen.
>

I have tested the xen_fbfront driver in linux stubdom. As shown in
attached file, I could print a small colored region.

[-- Attachment #2: Screenshot-QEMU (fedora14-dm)-4.png --]
[-- Type: image/png, Size: 8508 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Linux Stubdom Problem

[Stefano Stabellini]

On Tue, 23 Aug 2011, Jiageng Yu wrote:
> > I don't like the stubdom_vga_init approach: in general it is a good idea
> > to avoid state machines unless necessary.
> > I would implement a new function called xen_vga_ram_map in xen-all.c
> > that mmaps the xen_fbfront buffer and uses the new ioctl to
> > map the buffer into the guest.
> > xen_vga_ram_map should be called instead of xen_ram_alloc by
> > qemu_ram_alloc_from_ptr if name == "vga.vram".
> >
> 
> I have question here. I think xen_vga_ram_map() is used to instead of
> xc_map_foreign_bulk() which maps hvm guest's vram.
> 
> vga_common_init
>     -->qemu_get_ram_ptr
>              -->xen_map_cache
>                      -->xen_remap_bucket
>                                -->xc_map_foreign_bulk
> 
> The xen_ram_alloc() calls xc_domain_populate_physmap_exact() to
> increase the physical memory of hvm guest. And the increased physical
> memory becomes hvm guest's vram.
> 
> For the xen_remap_bucket(), there is no "vga.vram" input parameter. So
> we need the notification mechanism to invoke xen_vga_ram_map().

This is how I would do it:

vga_common_init
    -> qemu_ram_alloc
        -> qemu_ram_alloc_from_ptr
            -> xen_vga_ram_map (instead of xen_ram_alloc)
               the function also adds a locked mapcache entry that maps 
               offset to the mmap'ed framebuffer in qemu address space
    -> qemu_get_ram_ptr
        -> xen_map_cache should automatically return the mmap'ed
           framebuffer address thanks to the mapcache entry added by
           xen_vga_ram_map


Another problem is going to surface when cirrus_vga is going to try to
move the framebuffer (see cirrus_pci_lfb_map).
The xen specific function that ends up being called in this case is
xen_all.c:xen_add_to_physmap, we need to change it to call the new ioctl
rather then xc_domain_add_to_physmap.



> > Another suggestion: before starting to write any new lines of code, I
> > would make sure that mmaping the /dev/fb device actually works correctly.
> > For debugging purposes you can try to modify fbdev_init and write
> > something to the framebuffer right after the mmap, to see if the new
> > pattern is displayed correctly on the screen.
> >
> 
> I have tested the xen_fbfront driver in linux stubdom. As shown in
> attached file, I could print a small colored region.

Good!
Is everything working without a graphic card in the guest?
If so, we are really missing just the videoram bit.

Re: Re: Linux Stubdom Problem

[Tim Deegan]

At 20:36 +0100 on 22 Aug (1314045383), Stefano Stabellini wrote:
> The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
> mapping foreign mfns into the address space of the caller, while we need
> to remap a set of pages already mapped in the address space of the
> caller to some gmfns of a foreign domain. In other words we need the
> same functionality but in the other direction.

I think I was confused whan you told me about this yesterday.  Do you
need to map some of the linux-qemu-dom's memory into the guest?
In that case, I think you can just grant the pages to the guest VM and
use XENMAPSPACE_grant_table to map them. 

Cheers,

Tim.

-- 
Tim Deegan <tim@xen.org>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Tue, 23 Aug 2011, Tim Deegan wrote:
> At 20:36 +0100 on 22 Aug (1314045383), Stefano Stabellini wrote:
> > The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
> > mapping foreign mfns into the address space of the caller, while we need
> > to remap a set of pages already mapped in the address space of the
> > caller to some gmfns of a foreign domain. In other words we need the
> > same functionality but in the other direction.
> 
> I think I was confused whan you told me about this yesterday.  Do you
> need to map some of the linux-qemu-dom's memory into the guest?
> In that case, I think you can just grant the pages to the guest VM and
> use XENMAPSPACE_grant_table to map them. 

Yes, the pages I need to map into the guest have been vmalloc'ed by
xen_fbfront and mmap'ed into Qemu.
I didn't realize we could use any gmfn we wanted with
XENMAPSPACE_grant_table.

Jiageng: in the Linux implementation of the new ioctl, you can just call
gnttab_grant_foreign_access to grant access to the guest to framebuffer,
then you can use HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp) with
XENMAPSPACE_grant_table and the returned grant reference as idx to map
the pages into the guest.

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Tue, 23 Aug 2011, Stefano Stabellini wrote:
> On Tue, 23 Aug 2011, Tim Deegan wrote:
> > At 20:36 +0100 on 22 Aug (1314045383), Stefano Stabellini wrote:
> > > The current implementation of IOCTL_PRIVCMD_MMAPBATCH is only capable of
> > > mapping foreign mfns into the address space of the caller, while we need
> > > to remap a set of pages already mapped in the address space of the
> > > caller to some gmfns of a foreign domain. In other words we need the
> > > same functionality but in the other direction.
> > 
> > I think I was confused whan you told me about this yesterday.  Do you
> > need to map some of the linux-qemu-dom's memory into the guest?
> > In that case, I think you can just grant the pages to the guest VM and
> > use XENMAPSPACE_grant_table to map them. 
> 
> Yes, the pages I need to map into the guest have been vmalloc'ed by
> xen_fbfront and mmap'ed into Qemu.
> I didn't realize we could use any gmfn we wanted with
> XENMAPSPACE_grant_table.
> 
> Jiageng: in the Linux implementation of the new ioctl, you can just call
> gnttab_grant_foreign_access to grant access to the guest to framebuffer,
> then you can use HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp) with
> XENMAPSPACE_grant_table and the returned grant reference as idx to map
> the pages into the guest.
> 

It looks like XENMAPSPACE_grant_table is actually meant to map grant
table pages, meaning pages that are part of the grant table, not pages
that are referenced by the grant table.
So as it is we cannot use XENMAPSPACE_grant_table.

Unless you have another suggestion, or there are ref counting issues,
like Keir hinted, I think we should just go with implementing
XENMAPSPACE_mfn.

Re: Re: Linux Stubdom Problem

[Tim Deegan]

At 17:12 +0100 on 26 Aug (1314378720), Stefano Stabellini wrote:
> It looks like XENMAPSPACE_grant_table is actually meant to map grant
> table pages, meaning pages that are part of the grant table, not pages
> that are referenced by the grant table.

So it is; I had misremembered the interface.  GNTTABOP_map_grant_ref is 
the hypercall to map granted pages into the p2m. 

Tim.

-- 
Tim Deegan <tim@xen.org>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Sat, 27 Aug 2011, Tim Deegan wrote:
> At 17:12 +0100 on 26 Aug (1314378720), Stefano Stabellini wrote:
> > It looks like XENMAPSPACE_grant_table is actually meant to map grant
> > table pages, meaning pages that are part of the grant table, not pages
> > that are referenced by the grant table.
> 
> So it is; I had misremembered the interface.  GNTTABOP_map_grant_ref is 
> the hypercall to map granted pages into the p2m. 

Are you sure? I think that GNTTABOP_map_grant_ref maps foreign granted
pages into our own domain, in fact it takes a domid and a grant table
reference as input parameters to specify the source page, host_addr is
the destination in our own domain.

Re: Re: Linux Stubdom Problem

[Tim Deegan]

At 13:27 +0100 on 29 Aug (1314624464), Stefano Stabellini wrote:
> > So it is; I had misremembered the interface.  GNTTABOP_map_grant_ref is 
> > the hypercall to map granted pages into the p2m. 
> 
> Are you sure? I think that GNTTABOP_map_grant_ref maps foreign granted
> pages into our own domain, in fact it takes a domid and a grant table
> reference as input parameters to specify the source page, host_addr is
> the destination in our own domain.

Oh, so it will.  You'd need to arrange for that to be called from inside
the guest; or you could implement an add_to_physmap space for it; that
could be called from another domain.

Tim.

-- 
Tim Deegan <tim@xen.org>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Mon, 29 Aug 2011, Tim Deegan wrote:
> At 13:27 +0100 on 29 Aug (1314624464), Stefano Stabellini wrote:
> > > So it is; I had misremembered the interface.  GNTTABOP_map_grant_ref is 
> > > the hypercall to map granted pages into the p2m. 
> > 
> > Are you sure? I think that GNTTABOP_map_grant_ref maps foreign granted
> > pages into our own domain, in fact it takes a domid and a grant table
> > reference as input parameters to specify the source page, host_addr is
> > the destination in our own domain.
> 
> Oh, so it will.  You'd need to arrange for that to be called from inside
> the guest; or you could implement an add_to_physmap space for it; that
> could be called from another domain.

"From inside the guest" means hvmloader?
The good thing about doing it in hvmloader is that we could use the
traditional PV frontend/backend mechanism to share pages. On the other
hand hvmloader doesn't know if we are using stubdoms at the moment and
it would need to issue the grant table hypercall only in that case.
Unless we decide to always grant the videoram to guests but it would
change once again the domain to which the videoram is accounted for
(dom0/stubdom rather than the guest, that is a bad thing).
Also I don't like the idea of making hvmloader stubdom aware.

Thus I would choose the other option: implement an add_to_physmap space
for mapping grant references to another domain. Basically it would be
like what you suggested before (XENMAPSPACE_grant_table) but actually
mapping the granted pages rather than the pages constituting the grant
table :-)
However that means that with stubdoms the videoram is accounted to the
stubdom while without stubdoms the videoram is accounted to the guest.
I think we'll have to live with this inconsistency (the only other
solution would be to reimplement xen-fbfront in qemu).

Re: Re: Linux Stubdom Problem

[Keir Fraser]

On 29/08/2011 17:03, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
wrote:

>> Oh, so it will.  You'd need to arrange for that to be called from inside
>> the guest; or you could implement an add_to_physmap space for it; that
>> could be called from another domain.
> 
> "From inside the guest" means hvmloader?
> The good thing about doing it in hvmloader is that we could use the
> traditional PV frontend/backend mechanism to share pages. On the other
> hand hvmloader doesn't know if we are using stubdoms at the moment and
> it would need to issue the grant table hypercall only in that case.
> Unless we decide to always grant the videoram to guests but it would
> change once again the domain to which the videoram is accounted for
> (dom0/stubdom rather than the guest, that is a bad thing).
> Also I don't like the idea of making hvmloader stubdom aware.

I don't see a problem with it, in principle. I see hvmloader as almost an
in-guest part of the toolstack. The fact that it only executes at guest boot
means it can be fairly closely tied to the toolstack version.

 -- Keir

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

2011/8/31 Keir Fraser <keir.xen@gmail.com>:
> On 29/08/2011 17:03, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
> wrote:
>
>>> Oh, so it will.  You'd need to arrange for that to be called from inside
>>> the guest; or you could implement an add_to_physmap space for it; that
>>> could be called from another domain.
>>
>> "From inside the guest" means hvmloader?
>> The good thing about doing it in hvmloader is that we could use the
>> traditional PV frontend/backend mechanism to share pages. On the other
>> hand hvmloader doesn't know if we are using stubdoms at the moment and
>> it would need to issue the grant table hypercall only in that case.
>> Unless we decide to always grant the videoram to guests but it would
>> change once again the domain to which the videoram is accounted for
>> (dom0/stubdom rather than the guest, that is a bad thing).
>> Also I don't like the idea of making hvmloader stubdom aware.
>
> I don't see a problem with it, in principle. I see hvmloader as almost an
> in-guest part of the toolstack. The fact that it only executes at guest boot
> means it can be fairly closely tied to the toolstack version.
>
>  -- Keir
>
>
>

Hi all,

    I report a new issue about vram mapping in linux stubdom. I use
the follow patch to map the mfn of stubdom into hvm guest.

diff -r 0f36c2eec2e1 xen/arch/x86/mm.c
--- a/xen/arch/x86/mm.c	Thu Jul 28 15:40:54 2011 +0100
+++ b/xen/arch/x86/mm.c	Thu Sep 01 14:52:25 2011 +0100
@@ -4663,6 +4665,14 @@
             page = mfn_to_page(mfn);
             break;
         }
+       case XENMAPSPACE_mfn:
+	  {
+		if(!IS_PRIV_FOR(current->domain, d))
+			return -EINVAL;
+		mfn = xatp.idx;
+		page = mfn_to_page(mfn);
+		break;
+	  }
         default:
             break;
         }
@@ -4693,13 +4708,17 @@
         }

         /* Unmap from old location, if any. */
-        gpfn = get_gpfn_from_mfn(mfn);
-        ASSERT( gpfn != SHARED_M2P_ENTRY );
-        if ( gpfn != INVALID_M2P_ENTRY )
-            guest_physmap_remove_page(d, gpfn, mfn, 0);
+	 if(xatp.space!=XENMAPSPACE_mfn) {
+           gpfn = get_gpfn_from_mfn(mfn);
+           ASSERT( gpfn != SHARED_M2P_ENTRY );
+           if ( gpfn != INVALID_M2P_ENTRY )
+               guest_physmap_remove_page(d, gpfn, mfn, 0);
+	 }

         /* Map at new location. */
         rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);
diff -r 0f36c2eec2e1 xen/include/public/memory.h
--- a/xen/include/public/memory.h	Thu Jul 28 15:40:54 2011 +0100
+++ b/xen/include/public/memory.h	Thu Sep 01 14:52:25 2011 +0100
@@ -212,6 +212,7 @@
 #define XENMAPSPACE_shared_info 0 /* shared info page */
 #define XENMAPSPACE_grant_table 1 /* grant table page */
 #define XENMAPSPACE_gmfn        2 /* GMFN */
+#define XENMAPSPACE_mfn	    3 /* MFN  */
     unsigned int space;

 #define XENMAPIDX_grant_table_status 0x80000000


I got error at:

arch_memory_op()
   -->case XENMEM_add_to_physmap:
         -->if ( page )
              -->put_page(page);
                    -->free_domheap_page(page);
                           -->BUG_ON((pg[i].u.inuse.type_info &
PGT_count_mask) != 0);

In my case, pg[i].u.inuse.type_info & PGT_count_mask =1.

Actually, in the linux based stubdom case, I need to keep these pages
of vram mapped in qemu of stubdom. But it seems that granting pages
implies having the pages unmapped in the process that grants them.
Maybe the grant table could not solve the vram mapping problem.

Thanks,

Jiageng Yu.

Re: Re: Linux Stubdom Problem

[Tim Deegan]

At 01:12 +0800 on 02 Sep (1314925970), Jiageng Yu wrote:
> 2011/8/31 Keir Fraser <keir.xen@gmail.com>:
> > On 29/08/2011 17:03, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
> > wrote:
> >
> >>> Oh, so it will.  You'd need to arrange for that to be called from inside
> >>> the guest; or you could implement an add_to_physmap space for it; that
> >>> could be called from another domain.
> >>
> >> "From inside the guest" means hvmloader?
> >> The good thing about doing it in hvmloader is that we could use the
> >> traditional PV frontend/backend mechanism to share pages. On the other
> >> hand hvmloader doesn't know if we are using stubdoms at the moment and
> >> it would need to issue the grant table hypercall only in that case.
> >> Unless we decide to always grant the videoram to guests but it would
> >> change once again the domain to which the videoram is accounted for
> >> (dom0/stubdom rather than the guest, that is a bad thing).
> >> Also I don't like the idea of making hvmloader stubdom aware.
> >
> > I don't see a problem with it, in principle. I see hvmloader as almost an
> > in-guest part of the toolstack. The fact that it only executes at guest boot
> > means it can be fairly closely tied to the toolstack version.
> >
> >  -- Keir
> >
> >
> >
> 
> Hi all,
> 
>     I report a new issue about vram mapping in linux stubdom. I use
> the follow patch to map the mfn of stubdom into hvm guest.
> 
> diff -r 0f36c2eec2e1 xen/arch/x86/mm.c
> --- a/xen/arch/x86/mm.c	Thu Jul 28 15:40:54 2011 +0100
> +++ b/xen/arch/x86/mm.c	Thu Sep 01 14:52:25 2011 +0100
> @@ -4663,6 +4665,14 @@
>              page = mfn_to_page(mfn);
>              break;
>          }
> +       case XENMAPSPACE_mfn:
> +	  {
> +		if(!IS_PRIV_FOR(current->domain, d))
> +			return -EINVAL;
> +		mfn = xatp.idx;
> +		page = mfn_to_page(mfn);
> +		break;
> +	  }

I would really rather not have this interface; I don't see why we can't
use grant tables for this.

If you must do it this way, it should check that the MFN is valid and
that it's owned by the caller.

>          default:
>              break;
>          }
> @@ -4693,13 +4708,17 @@
>          }
> 
>          /* Unmap from old location, if any. */
> -        gpfn = get_gpfn_from_mfn(mfn);
> -        ASSERT( gpfn != SHARED_M2P_ENTRY );
> -        if ( gpfn != INVALID_M2P_ENTRY )
> -            guest_physmap_remove_page(d, gpfn, mfn, 0);
> +	 if(xatp.space!=XENMAPSPACE_mfn) {
> +           gpfn = get_gpfn_from_mfn(mfn);
> +           ASSERT( gpfn != SHARED_M2P_ENTRY );
> +           if ( gpfn != INVALID_M2P_ENTRY )
> +               guest_physmap_remove_page(d, gpfn, mfn, 0);
> +	 }

Why did you make this change?  

> 
>          /* Map at new location. */
>          rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);
> diff -r 0f36c2eec2e1 xen/include/public/memory.h
> --- a/xen/include/public/memory.h	Thu Jul 28 15:40:54 2011 +0100
> +++ b/xen/include/public/memory.h	Thu Sep 01 14:52:25 2011 +0100
> @@ -212,6 +212,7 @@
>  #define XENMAPSPACE_shared_info 0 /* shared info page */
>  #define XENMAPSPACE_grant_table 1 /* grant table page */
>  #define XENMAPSPACE_gmfn        2 /* GMFN */
> +#define XENMAPSPACE_mfn	    3 /* MFN  */
>      unsigned int space;
> 
>  #define XENMAPIDX_grant_table_status 0x80000000
> 
> 
> I got error at:
> 
> arch_memory_op()
>    -->case XENMEM_add_to_physmap:
>          -->if ( page )
>               -->put_page(page);
>                     -->free_domheap_page(page);
>                            -->BUG_ON((pg[i].u.inuse.type_info &
> PGT_count_mask) != 0);
> 
> In my case, pg[i].u.inuse.type_info & PGT_count_mask =1.

OK, so you've dropped the last untyped refcount on a page which still
has a type count.  That means the reference counting has got messed up
somewhere. 

> Actually, in the linux based stubdom case, I need to keep these pages
> of vram mapped in qemu of stubdom. But it seems that granting pages
> implies having the pages unmapped in the process that grants them.
> Maybe the grant table could not solve the vram mapping problem.

But this patch doesn't use the grant tables at all. 

Tim.

-- 

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

2011/9/2 Tim Deegan <tim@xen.org>:
> At 01:12 +0800 on 02 Sep (1314925970), Jiageng Yu wrote:
>> 2011/8/31 Keir Fraser <keir.xen@gmail.com>:
>> > On 29/08/2011 17:03, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
>> > wrote:
>> >
>> >>> Oh, so it will.  You'd need to arrange for that to be called from inside
>> >>> the guest; or you could implement an add_to_physmap space for it; that
>> >>> could be called from another domain.
>> >>
>> >> "From inside the guest" means hvmloader?
>> >> The good thing about doing it in hvmloader is that we could use the
>> >> traditional PV frontend/backend mechanism to share pages. On the other
>> >> hand hvmloader doesn't know if we are using stubdoms at the moment and
>> >> it would need to issue the grant table hypercall only in that case.
>> >> Unless we decide to always grant the videoram to guests but it would
>> >> change once again the domain to which the videoram is accounted for
>> >> (dom0/stubdom rather than the guest, that is a bad thing).
>> >> Also I don't like the idea of making hvmloader stubdom aware.
>> >
>> > I don't see a problem with it, in principle. I see hvmloader as almost an
>> > in-guest part of the toolstack. The fact that it only executes at guest boot
>> > means it can be fairly closely tied to the toolstack version.
>> >
>> >  -- Keir
>> >
>> >
>> >
>>
>> Hi all,
>>
>>     I report a new issue about vram mapping in linux stubdom. I use
>> the follow patch to map the mfn of stubdom into hvm guest.
>>
>> diff -r 0f36c2eec2e1 xen/arch/x86/mm.c
>> --- a/xen/arch/x86/mm.c       Thu Jul 28 15:40:54 2011 +0100
>> +++ b/xen/arch/x86/mm.c       Thu Sep 01 14:52:25 2011 +0100
>> @@ -4663,6 +4665,14 @@
>>              page = mfn_to_page(mfn);
>>              break;
>>          }
>> +       case XENMAPSPACE_mfn:
>> +       {
>> +             if(!IS_PRIV_FOR(current->domain, d))
>> +                     return -EINVAL;
>> +             mfn = xatp.idx;
>> +             page = mfn_to_page(mfn);
>> +             break;
>> +       }
>
> I would really rather not have this interface; I don't see why we can't
> use grant tables for this.
>

Hi,

    In linux based stubdom case, we want to keep hvm guest and its
hvmloader unaware of running on stubdom. Therefore, we do need a way
to map vram pages of stubdom into guest hvm transparently.

   If we use the grant tables for this, the general procedure would be:

   1.  stubdom allocates vram pages
   2.  stubdom creates the grant reference for these pages and
transfers the GRs to hvm guest(or hvmloader)
   3.  hvm guest(or hvmloader) maps these pages into its memory space
using the GRs
   4.  stubdom and hvm guest share the same vram memory.

   In this procedure, the hvm guest(or hvmloader) must know it is
running on stubdom.

   Additionally, if I modified grant table to map pages without any
participation of hvm guest(or hvmloader), it will obey the design
goals of grant table. So I think grant table may not be suitable for
our case.

   Another idea is to allocate vram in hvm guest and stubdom maps vram
pages into its memory space. In this scenario, I must delay the
initial process of xen-fbfront in stubdom until qemu populates vram
pages for hvm guest (unsing xc_domain_populate_physmap_exact()). Then,
I map these pages into stubdom kernel through the function similar to
privcmd_ioctl_mmap_batch(). But I would not use privcmd device.

   If anyone has any advice for me I'm all ears.

   Thanks.

Jiageng Yu.

> If you must do it this way, it should check that the MFN is valid and
> that it's owned by the caller.
>
>>          default:
>>              break;
>>          }
>> @@ -4693,13 +4708,17 @@
>>          }
>>
>>          /* Unmap from old location, if any. */
>> -        gpfn = get_gpfn_from_mfn(mfn);
>> -        ASSERT( gpfn != SHARED_M2P_ENTRY );
>> -        if ( gpfn != INVALID_M2P_ENTRY )
>> -            guest_physmap_remove_page(d, gpfn, mfn, 0);
>> +      if(xatp.space!=XENMAPSPACE_mfn) {
>> +           gpfn = get_gpfn_from_mfn(mfn);
>> +           ASSERT( gpfn != SHARED_M2P_ENTRY );
>> +           if ( gpfn != INVALID_M2P_ENTRY )
>> +               guest_physmap_remove_page(d, gpfn, mfn, 0);
>> +      }
>
> Why did you make this change?
>
>>
>>          /* Map at new location. */
>>          rc = guest_physmap_add_page(d, xatp.gpfn, mfn, 0);
>> diff -r 0f36c2eec2e1 xen/include/public/memory.h
>> --- a/xen/include/public/memory.h     Thu Jul 28 15:40:54 2011 +0100
>> +++ b/xen/include/public/memory.h     Thu Sep 01 14:52:25 2011 +0100
>> @@ -212,6 +212,7 @@
>>  #define XENMAPSPACE_shared_info 0 /* shared info page */
>>  #define XENMAPSPACE_grant_table 1 /* grant table page */
>>  #define XENMAPSPACE_gmfn        2 /* GMFN */
>> +#define XENMAPSPACE_mfn          3 /* MFN  */
>>      unsigned int space;
>>
>>  #define XENMAPIDX_grant_table_status 0x80000000
>>
>>
>> I got error at:
>>
>> arch_memory_op()
>>    -->case XENMEM_add_to_physmap:
>>          -->if ( page )
>>               -->put_page(page);
>>                     -->free_domheap_page(page);
>>                            -->BUG_ON((pg[i].u.inuse.type_info &
>> PGT_count_mask) != 0);
>>
>> In my case, pg[i].u.inuse.type_info & PGT_count_mask =1.
>
> OK, so you've dropped the last untyped refcount on a page which still
> has a type count.  That means the reference counting has got messed up
> somewhere.
>
>> Actually, in the linux based stubdom case, I need to keep these pages
>> of vram mapped in qemu of stubdom. But it seems that granting pages
>> implies having the pages unmapped in the process that grants them.
>> Maybe the grant table could not solve the vram mapping problem.
>
> But this patch doesn't use the grant tables at all.
>
> Tim.
>
> --
>

Re: Re: Linux Stubdom Problem

[Tim Deegan]

Hi, 

At 10:32 +0800 on 02 Sep (1314959538), Jiageng Yu wrote:
> 2011/9/2 Tim Deegan <tim@xen.org>:
> > I would really rather not have this interface; I don't see why we can't
> > use grant tables for this.
> 
>     In linux based stubdom case, we want to keep hvm guest and its
> hvmloader unaware of running on stubdom.

Why?  HVMloader is already tightly coupled to the hypervisor and the
toostack - special cases for stubdoms should be fine.

> Therefore, we do need a way
> to map vram pages of stubdom into guest hvm transparently.

I've suggested two so far: have grant mappings done from inside the
guest, or add a XENMAPSPACE that takes grant IDs.  I think the
XENMAPSPACE is better; I suspect that save/restore will be easier to get
right that way.

>    Additionally, if I modified grant table to map pages without any
> participation of hvm guest(or hvmloader), it will obey the design
> goals of grant table. So I think grant table may not be suitable for
> our case.

I don't understand you.

>    Another idea is to allocate vram in hvm guest and stubdom maps vram
> pages into its memory space.

Sure.  The minios-based stubdoms seem to manage that just fine.  If this
is really difficult for a linux-based stub domain, then maybe that's a
reason not to use them.

Cheers,

Tim.

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Fri, 2 Sep 2011, Tim Deegan wrote:
> At 10:32 +0800 on 02 Sep (1314959538), Jiageng Yu wrote:
> > 2011/9/2 Tim Deegan <tim@xen.org>:
> > > I would really rather not have this interface; I don't see why we can't
> > > use grant tables for this.
> > 
> >     In linux based stubdom case, we want to keep hvm guest and its
> > hvmloader unaware of running on stubdom.
> 
> Why?  HVMloader is already tightly coupled to the hypervisor and the
> toostack - special cases for stubdoms should be fine.

I think think that leaking the implementation details of the device
model into hvmloader should be avoided, but obviously if there are no
alternatives, it can be done.


> > Therefore, we do need a way
> > to map vram pages of stubdom into guest hvm transparently.
> 
> I've suggested two so far: have grant mappings done from inside the
> guest, or add a XENMAPSPACE that takes grant IDs.  I think the
> XENMAPSPACE is better; I suspect that save/restore will be easier to get
> right that way.

OK. I think we'll try the other approach first to see if it is easier:
modify Linux xen-fbfront driver to take a list of pages from the guest
for the vram.


> >    Another idea is to allocate vram in hvm guest and stubdom maps vram
> > pages into its memory space.
> 
> Sure.  The minios-based stubdoms seem to manage that just fine.  If this
> is really difficult for a linux-based stub domain, then maybe that's a
> reason not to use them.

We could fully re-implement xen-fbfront in userspace inside qemu, at
that point the problem would go away completely.
Rather than duplicating all that code, we'll try to reuse Linux
xen-fbfront implementation, making sure that xen-fbfront is loaded after
qemu is started and initialized.

Re: Re: Linux Stubdom Problem

[Keir Fraser]

On 02/09/2011 14:09, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
wrote:

>> Why?  HVMloader is already tightly coupled to the hypervisor and the
>> toostack - special cases for stubdoms should be fine.
> 
> I think think that leaking the implementation details of the device
> model into hvmloader should be avoided, but obviously if there are no
> alternatives, it can be done.

This is a fair and more general point, that we don't want fragile
dependencies on qemu now that we are using upstream. But as I say that's a
more general point on our policy regarding qemu, rather than something
specifically concerning hvmloader.

 -- Keir

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

2011/9/2 Keir Fraser <keir.xen@gmail.com>:
> On 02/09/2011 14:09, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com>
> wrote:
>
>>> Why?  HVMloader is already tightly coupled to the hypervisor and the
>>> toostack - special cases for stubdoms should be fine.
>>
>> I think think that leaking the implementation details of the device
>> model into hvmloader should be avoided, but obviously if there are no
>> alternatives, it can be done.
>
> This is a fair and more general point, that we don't want fragile
> dependencies on qemu now that we are using upstream. But as I say that's a
> more general point on our policy regarding qemu, rather than something
> specifically concerning hvmloader.
>
>  -- Keir
>
>
>

Hi Stefano,

     I just have a prototype of vram mapping and test it now. The
implementation of linux-stubdom kernel part is as follows.
xen_remap_domain_mfn_range2 function maps foreign dom's physical
address into linux kernel space. It is similar to
xen_remap_domain_mfn_range. But xen_remap_domain_mfn_range is used to
map foreign pages into linux user space.

    But the page info seems wrong after executing xen_remap_domain_mfn_range2.

    struct page *page=pfn_to_page(vmalloc_to_pfn(info->fb));

    The page->_count = 0xc2c2c2c2. It is very strange.

    Did I do the right thing?

    Greeting.

Jiageng Yu.


diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 204e3ba..72a7808 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2693,6 +2693,73 @@ out:
 }
 EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);

+int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long gpfn,
+                int nr, unsigned domid)
+{
+    struct remap_data rmd;
+    struct mmu_update mmu_update[REMAP_BATCH_SIZE];
+    int level,i,batch,nr_page = nr;
+    unsigned long range;
+    int err = 0;
+    unsigned long vaddr,base_addr = addr;
+    pte_t pte,*ptep;
+
+    rmd.mfn = gpfn;
+    rmd.prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |
_PAGE_IOMAP);
+
+    while(nr_page) {
+        batch = min(REMAP_BATCH_SIZE, nr);
+        range = (unsigned long)batch << PAGE_SHIFT;
+
+        rmd.mmu_update = mmu_update;
+
+        for(i=0; i < batch; i++){
+            pte = pte_mkspecial(pfn_pte(rmd.mfn++, rmd.prot));
+            vaddr = base_addr + i*PAGE_SIZE;
+            ptep = lookup_address(vaddr, &level);
+
+            rmd.mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr |
+                                        MMU_NORMAL_PT_UPDATE;
+            rmd.mmu_update->val = pte_val_ma(pte);
+            rmd.mmu_update++;
+        }
+
+        err = -EFAULT;
+        if(HYPERVISOR_mmu_update(mmu_update, batch, NULL, domid) < 0)
+            goto out;
+
+        nr_page -= batch;
+        base_addr += range;
+    }
+
+    err = 0;
+
+    base_addr = addr;
+    for(i=0; i < nr; i++){
+        vaddr = base_addr + i*PAGE_SIZE;
+        set_phys_to_machine(vmalloc_to_pfn(vaddr),
+                arbitrary_virt_to_machine(vaddr).maddr >> PAGE_SHIFT);
+    }
+
+out:
+	flush_tlb_all();
+	return err;
+}
+EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range2);
 #ifdef CONFIG_XEN_PVHVM
 static void xen_hvm_exit_mmap(struct mm_struct *mm)
 {
diff --git a/drivers/video/xen-fbfront.c b/drivers/video/xen-fbfront.c
index dc72563..82da2ee 100644
--- a/drivers/video/xen-fbfront.c
+++ b/drivers/video/xen-fbfront.c
@@ -25,8 +25,12 @@
 #include <linux/module.h>
 #include <linux/vmalloc.h>
 #include <linux/mm.h>
+#include <linux/sched.h>
+#include <asm/pgtable.h>
+#include <asm/page.h>

 #include <asm/xen/hypervisor.h>
+#include <asm/xen/page.h>

 #include <xen/xen.h>
 #include <xen/events.h>
@@ -34,6 +38,7 @@
 #include <xen/interface/io/fbif.h>
 #include <xen/interface/io/protocols.h>
 #include <xen/xenbus.h>
+#include <xen/xen-ops.h>

 struct xenfb_info {
 	unsigned char		*fb;
@@ -62,6 +67,12 @@ module_param_array(video, int, NULL, 0);
 MODULE_PARM_DESC(video,
 	"Video memory size in MB, width, height in pixels (default 2,800,600)");

+static unsigned long foreign_vaddr = 0;
+module_param(foreign_vaddr, ulong, S_IRUGO);
+
+static unsigned long foreign_domid = 0;
+module_param(foreign_domid, ulong, S_IRUGO);
+
 static void xenfb_make_preferred_console(void);
 static int xenfb_remove(struct xenbus_device *);
 static void xenfb_init_shared_page(struct xenfb_info *, struct fb_info *);
@@ -398,7 +408,17 @@ static int __devinit xenfb_probe(struct xenbus_device *dev,
 	if (info->fb == NULL)
 		goto error_nomem;
 	memset(info->fb, 0, fb_size);
-
+    if((foreign_vaddr != 0) && (foreign_domid != 0)){
+        ret = xen_remap_domain_mfn_range2((unsigned long)(info->fb),
+                                    foreign_vaddr >> PAGE_SHIFT,
+                                    fb_size >> PAGE_SHIFT, foreign_domid);
+        if(ret < 0){
+            printk("Can not remap vram of hvm guest.\n");
+            goto error;
+        }
+    }
 	info->nr_pages = (fb_size + PAGE_SIZE - 1) >> PAGE_SHIFT;

 	info->mfns = vmalloc(sizeof(unsigned long) * info->nr_pages);
diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
index 4349e89..1554531 100644
--- a/include/xen/xen-ops.h
+++ b/include/xen/xen-ops.h
@@ -20,6 +20,10 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
 			       unsigned long mfn, int nr,
 			       pgprot_t prot, unsigned domid);

+int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long mfn,
+                int nr, unsigned domid);
 extern unsigned long *xen_contiguous_bitmap;
 int xen_create_contiguous_region(unsigned long vstart, unsigned int order,
 				unsigned int address_bits);

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Wed, 14 Sep 2011, Jiageng Yu wrote:
> Hi Stefano,
> 
>      I just have a prototype of vram mapping and test it now. The
> implementation of linux-stubdom kernel part is as follows.
> xen_remap_domain_mfn_range2 function maps foreign dom's physical
> address into linux kernel space. It is similar to
> xen_remap_domain_mfn_range. But xen_remap_domain_mfn_range is used to
> map foreign pages into linux user space.
> 
>     But the page info seems wrong after executing xen_remap_domain_mfn_range2.
> 
>     struct page *page=pfn_to_page(vmalloc_to_pfn(info->fb));
> 
>     The page->_count = 0xc2c2c2c2. It is very strange.
> 
>     Did I do the right thing?
> 

use page_address instead of pfn_to_page to find the struct page


>     Greeting.
> 
> Jiageng Yu.
> 
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 204e3ba..72a7808 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -2693,6 +2693,73 @@ out:
>  }
>  EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
> 
> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long gpfn,
> +                int nr, unsigned domid)
> +{
> +    struct remap_data rmd;
> +    struct mmu_update mmu_update[REMAP_BATCH_SIZE];
> +    int level,i,batch,nr_page = nr;
> +    unsigned long range;
> +    int err = 0;
> +    unsigned long vaddr,base_addr = addr;
> +    pte_t pte,*ptep;
> +
> +    rmd.mfn = gpfn;
> +    rmd.prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |
> _PAGE_IOMAP);
> +
> +    while(nr_page) {
> +        batch = min(REMAP_BATCH_SIZE, nr);
> +        range = (unsigned long)batch << PAGE_SHIFT;
> +
> +        rmd.mmu_update = mmu_update;
> +
> +        for(i=0; i < batch; i++){
> +            pte = pte_mkspecial(pfn_pte(rmd.mfn++, rmd.prot));
> +            vaddr = base_addr + i*PAGE_SIZE;
> +            ptep = lookup_address(vaddr, &level);

you need to check if ptep is valid here and the level is PG_LEVEL_4K

> +            rmd.mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr |
> +                                        MMU_NORMAL_PT_UPDATE;

you can use pte_mfn(*ptep) instead of arbitrary_virt_to_machine


> +            rmd.mmu_update->val = pte_val_ma(pte);
> +            rmd.mmu_update++;
> +        }
> +
> +        err = -EFAULT;
> +        if(HYPERVISOR_mmu_update(mmu_update, batch, NULL, domid) < 0)
> +            goto out;
> +
> +        nr_page -= batch;
> +        base_addr += range;
> +    }
> +
> +    err = 0;
> +
> +    base_addr = addr;
> +    for(i=0; i < nr; i++){
> +        vaddr = base_addr + i*PAGE_SIZE;
> +        set_phys_to_machine(vmalloc_to_pfn(vaddr),
> +                arbitrary_virt_to_machine(vaddr).maddr >> PAGE_SHIFT);
> +    }

The second argument (mfn) to set_phys_to_machine is wrong:
arbitrary_virt_to_machine ends up calling virt_to_machine if
virt_addr_valid. You need to manually call pte_mfn:

/* the ptep content has been updated by Xen so we can lookup the foreign
 * mfn from the pte now */
pte = lookup_address(vaddr, &level);
BUG_ON(pte == NULL);
offset = vaddr & ~PAGE_MASK;
mfn = XMADDR(((phys_addr_t)pte_mfn(*pte) << PAGE_SHIFT) + offset);


> +
> +out:
> +	flush_tlb_all();
> +	return err;
> +}
> +EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range2);

the name should be changed to xen_remap_foreign_gpfn_range


>  #ifdef CONFIG_XEN_PVHVM
>  static void xen_hvm_exit_mmap(struct mm_struct *mm)
>  {
> diff --git a/drivers/video/xen-fbfront.c b/drivers/video/xen-fbfront.c
> index dc72563..82da2ee 100644
> --- a/drivers/video/xen-fbfront.c
> +++ b/drivers/video/xen-fbfront.c
> @@ -25,8 +25,12 @@
>  #include <linux/module.h>
>  #include <linux/vmalloc.h>
>  #include <linux/mm.h>
> +#include <linux/sched.h>
> +#include <asm/pgtable.h>
> +#include <asm/page.h>
> 
>  #include <asm/xen/hypervisor.h>
> +#include <asm/xen/page.h>
> 
>  #include <xen/xen.h>
>  #include <xen/events.h>
> @@ -34,6 +38,7 @@
>  #include <xen/interface/io/fbif.h>
>  #include <xen/interface/io/protocols.h>
>  #include <xen/xenbus.h>
> +#include <xen/xen-ops.h>
> 
>  struct xenfb_info {
>  	unsigned char		*fb;
> @@ -62,6 +67,12 @@ module_param_array(video, int, NULL, 0);
>  MODULE_PARM_DESC(video,
>  	"Video memory size in MB, width, height in pixels (default 2,800,600)");
> 
> +static unsigned long foreign_vaddr = 0;
> +module_param(foreign_vaddr, ulong, S_IRUGO);
> +
> +static unsigned long foreign_domid = 0;
> +module_param(foreign_domid, ulong, S_IRUGO);
> +
>  static void xenfb_make_preferred_console(void);
>  static int xenfb_remove(struct xenbus_device *);
>  static void xenfb_init_shared_page(struct xenfb_info *, struct fb_info *);
> @@ -398,7 +408,17 @@ static int __devinit xenfb_probe(struct xenbus_device *dev,
>  	if (info->fb == NULL)
>  		goto error_nomem;
>  	memset(info->fb, 0, fb_size);
> -
> +    if((foreign_vaddr != 0) && (foreign_domid != 0)){
> +        ret = xen_remap_domain_mfn_range2((unsigned long)(info->fb),
> +                                    foreign_vaddr >> PAGE_SHIFT,
> +                                    fb_size >> PAGE_SHIFT, foreign_domid);

you should rename foreign_vaddr to foreign_gpfn and pass the gpfn value
that is the ram_addr (page shifted) passed to xen_ram_alloc in qemu.

> +        if(ret < 0){
> +            printk("Can not remap vram of hvm guest.\n");
> +            goto error;
> +        }
> +    }
>  	info->nr_pages = (fb_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
> 
>  	info->mfns = vmalloc(sizeof(unsigned long) * info->nr_pages);
> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
> index 4349e89..1554531 100644
> --- a/include/xen/xen-ops.h
> +++ b/include/xen/xen-ops.h
> @@ -20,6 +20,10 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
>  			       unsigned long mfn, int nr,
>  			       pgprot_t prot, unsigned domid);
> 
> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long mfn,
> +                int nr, unsigned domid);
>  extern unsigned long *xen_contiguous_bitmap;
>  int xen_create_contiguous_region(unsigned long vstart, unsigned int order,
>  				unsigned int address_bits);
> 

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1: Type: text/plain, Size: 8773 bytes --]

2011/9/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Wed, 14 Sep 2011, Jiageng Yu wrote:
>> Hi Stefano,
>>
>>      I just have a prototype of vram mapping and test it now. The
>> implementation of linux-stubdom kernel part is as follows.
>> xen_remap_domain_mfn_range2 function maps foreign dom's physical
>> address into linux kernel space. It is similar to
>> xen_remap_domain_mfn_range. But xen_remap_domain_mfn_range is used to
>> map foreign pages into linux user space.
>>
>>     But the page info seems wrong after executing xen_remap_domain_mfn_range2.
>>
>>     struct page *page=pfn_to_page(vmalloc_to_pfn(info->fb));
>>
>>     The page->_count = 0xc2c2c2c2. It is very strange.
>>
>>     Did I do the right thing?
>>
>
> use page_address instead of pfn_to_page to find the struct page
>
>
>>     Greeting.
>>
>> Jiageng Yu.
>>
>>
>> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
>> index 204e3ba..72a7808 100644
>> --- a/arch/x86/xen/mmu.c
>> +++ b/arch/x86/xen/mmu.c
>> @@ -2693,6 +2693,73 @@ out:
>>  }
>>  EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
>>
>> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long gpfn,
>> +                int nr, unsigned domid)
>> +{
>> +    struct remap_data rmd;
>> +    struct mmu_update mmu_update[REMAP_BATCH_SIZE];
>> +    int level,i,batch,nr_page = nr;
>> +    unsigned long range;
>> +    int err = 0;
>> +    unsigned long vaddr,base_addr = addr;
>> +    pte_t pte,*ptep;
>> +
>> +    rmd.mfn = gpfn;
>> +    rmd.prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |
>> _PAGE_IOMAP);
>> +
>> +    while(nr_page) {
>> +        batch = min(REMAP_BATCH_SIZE, nr);
>> +        range = (unsigned long)batch << PAGE_SHIFT;
>> +
>> +        rmd.mmu_update = mmu_update;
>> +
>> +        for(i=0; i < batch; i++){
>> +            pte = pte_mkspecial(pfn_pte(rmd.mfn++, rmd.prot));
>> +            vaddr = base_addr + i*PAGE_SIZE;
>> +            ptep = lookup_address(vaddr, &level);
>
> you need to check if ptep is valid here and the level is PG_LEVEL_4K
>
>> +            rmd.mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr |
>> +                                        MMU_NORMAL_PT_UPDATE;
>
> you can use pte_mfn(*ptep) instead of arbitrary_virt_to_machine
>
>
>> +            rmd.mmu_update->val = pte_val_ma(pte);
>> +            rmd.mmu_update++;
>> +        }
>> +
>> +        err = -EFAULT;
>> +        if(HYPERVISOR_mmu_update(mmu_update, batch, NULL, domid) < 0)
>> +            goto out;
>> +
>> +        nr_page -= batch;
>> +        base_addr += range;
>> +    }
>> +
>> +    err = 0;
>> +
>> +    base_addr = addr;
>> +    for(i=0; i < nr; i++){
>> +        vaddr = base_addr + i*PAGE_SIZE;
>> +        set_phys_to_machine(vmalloc_to_pfn(vaddr),
>> +                arbitrary_virt_to_machine(vaddr).maddr >> PAGE_SHIFT);
>> +    }
>
> The second argument (mfn) to set_phys_to_machine is wrong:
> arbitrary_virt_to_machine ends up calling virt_to_machine if
> virt_addr_valid. You need to manually call pte_mfn:
>
> /* the ptep content has been updated by Xen so we can lookup the foreign
>  * mfn from the pte now */
> pte = lookup_address(vaddr, &level);
> BUG_ON(pte == NULL);
> offset = vaddr & ~PAGE_MASK;
> mfn = XMADDR(((phys_addr_t)pte_mfn(*pte) << PAGE_SHIFT) + offset);
>
>
>> +
>> +out:
>> +     flush_tlb_all();
>> +     return err;
>> +}
>> +EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range2);
>
> the name should be changed to xen_remap_foreign_gpfn_range
>
>
>>  #ifdef CONFIG_XEN_PVHVM
>>  static void xen_hvm_exit_mmap(struct mm_struct *mm)
>>  {
>> diff --git a/drivers/video/xen-fbfront.c b/drivers/video/xen-fbfront.c
>> index dc72563..82da2ee 100644
>> --- a/drivers/video/xen-fbfront.c
>> +++ b/drivers/video/xen-fbfront.c
>> @@ -25,8 +25,12 @@
>>  #include <linux/module.h>
>>  #include <linux/vmalloc.h>
>>  #include <linux/mm.h>
>> +#include <linux/sched.h>
>> +#include <asm/pgtable.h>
>> +#include <asm/page.h>
>>
>>  #include <asm/xen/hypervisor.h>
>> +#include <asm/xen/page.h>
>>
>>  #include <xen/xen.h>
>>  #include <xen/events.h>
>> @@ -34,6 +38,7 @@
>>  #include <xen/interface/io/fbif.h>
>>  #include <xen/interface/io/protocols.h>
>>  #include <xen/xenbus.h>
>> +#include <xen/xen-ops.h>
>>
>>  struct xenfb_info {
>>       unsigned char           *fb;
>> @@ -62,6 +67,12 @@ module_param_array(video, int, NULL, 0);
>>  MODULE_PARM_DESC(video,
>>       "Video memory size in MB, width, height in pixels (default 2,800,600)");
>>
>> +static unsigned long foreign_vaddr = 0;
>> +module_param(foreign_vaddr, ulong, S_IRUGO);
>> +
>> +static unsigned long foreign_domid = 0;
>> +module_param(foreign_domid, ulong, S_IRUGO);
>> +
>>  static void xenfb_make_preferred_console(void);
>>  static int xenfb_remove(struct xenbus_device *);
>>  static void xenfb_init_shared_page(struct xenfb_info *, struct fb_info *);
>> @@ -398,7 +408,17 @@ static int __devinit xenfb_probe(struct xenbus_device *dev,
>>       if (info->fb == NULL)
>>               goto error_nomem;
>>       memset(info->fb, 0, fb_size);
>> -
>> +    if((foreign_vaddr != 0) && (foreign_domid != 0)){
>> +        ret = xen_remap_domain_mfn_range2((unsigned long)(info->fb),
>> +                                    foreign_vaddr >> PAGE_SHIFT,
>> +                                    fb_size >> PAGE_SHIFT, foreign_domid);
>
> you should rename foreign_vaddr to foreign_gpfn and pass the gpfn value
> that is the ram_addr (page shifted) passed to xen_ram_alloc in qemu.
>
>> +        if(ret < 0){
>> +            printk("Can not remap vram of hvm guest.\n");
>> +            goto error;
>> +        }
>> +    }
>>       info->nr_pages = (fb_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
>>
>>       info->mfns = vmalloc(sizeof(unsigned long) * info->nr_pages);
>> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
>> index 4349e89..1554531 100644
>> --- a/include/xen/xen-ops.h
>> +++ b/include/xen/xen-ops.h
>> @@ -20,6 +20,10 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
>>                              unsigned long mfn, int nr,
>>                              pgprot_t prot, unsigned domid);
>>
>> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long mfn,
>> +                int nr, unsigned domid);
>>  extern unsigned long *xen_contiguous_bitmap;
>>  int xen_create_contiguous_region(unsigned long vstart, unsigned int order,
>>                               unsigned int address_bits);
>>
>


Hi Stefano,

	I have some progress in linux based stubdom project. As shown in the
attached video, I have started the emulated vga device in the linux
based stubdom.

	In a short conclusion, for the linux based stubdom, there are two
major problems about vga device emulation. The first is the vram
mapping, which we discussed a lot previously and handled it. Another
is the vga BIOS mapping (address 0xc0000-0xc8fff of hvm guest).

	I found the vga BIOS mapping problem in remap_area_mfn_pte_fn()
function. The pte_mkspecial() will return invalid value when I try to
map 0xc0000-0xc8fff into linux based stubdom.

pte_mkspecial()
	->pte_set_flags()
		->native_pte_val()
		->native_make_pte()

	According to my test, the root cause of vga BIOS mapping problem is
native_xxx functions. We could avoid the problem by invoking functions
defined in paravirt.h instead. The patch is as follows. But I think it
is not a good way to handle the problem. Maybe you can give me some
suggestions.

	I also found the hard disk didn’t work well. I will investigate it these days.


--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2639,12 +2640,16 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
pgtable_t token,
 				 unsigned long addr, void *data)
 {
 	struct remap_data *rmd = data;
-	pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
+    if((rmd->mfn & 0xfffffff0) == 0xc0){
+	    pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
+	    rmd->mmu_update->val = pte_val(pte);
+    }else{
+	    pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
+	    rmd->mmu_update->val = pte_val_ma(pte);
+    }

 	rmd->mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr;
-	rmd->mmu_update->val = pte_val_ma(pte);
 	rmd->mmu_update++;
-
 	return 0;
 }

[-- Attachment #2: out.ogv --]
[-- Type: video/ogg, Size: 3783903 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3802 bytes --]

On Thu, 27 Oct 2011, Jiageng Yu wrote:
> Hi Stefano,
> 
> 	I have some progress in linux based stubdom project. As shown in the
> attached video, I have started the emulated vga device in the linux
> based stubdom.
> 
> 	In a short conclusion, for the linux based stubdom, there are two
> major problems about vga device emulation. The first is the vram
> mapping, which we discussed a lot previously and handled it.

Do you have an updated version of the patch you used?


> Another
> is the vga BIOS mapping (address 0xc0000-0xc8fff of hvm guest).

This is caused by qemu trying to map that memory area in its own address
space, right?


> 	I found the vga BIOS mapping problem in remap_area_mfn_pte_fn()
> function. The pte_mkspecial() will return invalid value when I try to
> map 0xc0000-0xc8fff into linux based stubdom.

What is exactly the error you are seeing?


> pte_mkspecial()
> 	->pte_set_flags()
> 		->native_pte_val()
> 		->native_make_pte()
> 
> 	According to my test, the root cause of vga BIOS mapping problem is
> native_xxx functions. We could avoid the problem by invoking functions
> defined in paravirt.h instead. The patch is as follows. But I think it
> is not a good way to handle the problem. Maybe you can give me some
> suggestions.
> 
> 	I also found the hard disk didn�?�????�?�???�?�??�?�?�?�¢??t work well. I will investigate it these days.
> 
> 
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -2639,12 +2640,16 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
> pgtable_t token,
>  				 unsigned long addr, void *data)
>  {
>  	struct remap_data *rmd = data;
> -	pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> +    if((rmd->mfn & 0xfffffff0) == 0xc0){
> +	    pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
> +	    rmd->mmu_update->val = pte_val(pte);
> +    }else{
> +	    pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> +	    rmd->mmu_update->val = pte_val_ma(pte);
> +    }

Even if the fix is not the correct one I think I might understand what
the real problem is:

pfn_pte -> xen_make_pte

if (unlikely(pte & _PAGE_IOMAP) &&
        (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
    pte = iomap_pte(pte);
} else {
    pte &= ~_PAGE_IOMAP;
    pte = pte_pfn_to_mfn(pte);
}

considering that in this case xen_initial_domain() returns false and
addr is < ISA_END_ADDRESS (it is a gpfn address), xen_make_pte is going
to threat the mfn as a pfn erroneously.

In your patch you replaced pte_val_ma with pte_val that does the
opposite translation (mfn -> pfn) so the end result is that you get the
original mfn in rmd->mmu_update->val.

The real fix should something along these lines:



diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 3dd53f9..f2fadfc 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
 		pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
 	}
 
-	if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
+	if (pteval & _PAGE_IOMAP)
 		return pteval;
 
 	return pte_mfn_to_pfn(pteval);
@@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
 	 * mappings are just dummy local mappings to keep other
 	 * parts of the kernel happy.
 	 */
-	if (unlikely(pte & _PAGE_IOMAP) &&
-	    (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
+	if (unlikely(pte & _PAGE_IOMAP)) {
 		pte = iomap_pte(pte);
 	} else {
 		pte &= ~_PAGE_IOMAP;
---

Could you please confirm whether this patch fixes your problem?

Konrad, do you know if this could have any unintended consequences?
I don't think it can be a problem security wise because Xen is going to
do all the permission checks anyway.
The only problem I can see is if a domU is going to call xen_make_pte
with _PAGE_IOMAP and a pfn->mfn translation is supposed to happen.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

2011/11/9 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>
> On Thu, 27 Oct 2011, Jiageng Yu wrote:
> > Hi Stefano,
> >
> >       I have some progress in linux based stubdom project. As shown in the
> > attached video, I have started the emulated vga device in the linux
> > based stubdom.
> >
> >       In a short conclusion, for the linux based stubdom, there are two
> > major problems about vga device emulation. The first is the vram
> > mapping, which we discussed a lot previously and handled it.
>
> Do you have an updated version of the patch you used?
>
>
> > Another
> > is the vga BIOS mapping (address 0xc0000-0xc8fff of hvm guest).
>
> This is caused by qemu trying to map that memory area in its own address
> space, right?
>
>
> >       I found the vga BIOS mapping problem in remap_area_mfn_pte_fn()
> > function. The pte_mkspecial() will return invalid value when I try to
> > map 0xc0000-0xc8fff into linux based stubdom.
>
> What is exactly the error you are seeing?
>
>
> > pte_mkspecial()
> >       ->pte_set_flags()
> >               ->native_pte_val()
> >               ->native_make_pte()
> >
> >       According to my test, the root cause of vga BIOS mapping problem is
> > native_xxx functions. We could avoid the problem by invoking functions
> > defined in paravirt.h instead. The patch is as follows. But I think it
> > is not a good way to handle the problem. Maybe you can give me some
> > suggestions.
> >
> >       I also found the hard disk didnÃ??Ã?????Ã??Ã????Ã??Ã???Ã??Ã??Ã??Ã?¢??t work well. I will investigate it these days.
> >
> >
> > --- a/arch/x86/xen/mmu.c
> > +++ b/arch/x86/xen/mmu.c
> > @@ -2639,12 +2640,16 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
> > pgtable_t token,
> >                                unsigned long addr, void *data)
> >  {
> >       struct remap_data *rmd = data;
> > -     pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> > +    if((rmd->mfn & 0xfffffff0) == 0xc0){
> > +         pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
> > +         rmd->mmu_update->val = pte_val(pte);
> > +    }else{
> > +         pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> > +         rmd->mmu_update->val = pte_val_ma(pte);
> > +    }
>
> Even if the fix is not the correct one I think I might understand what
> the real problem is:
>
> pfn_pte -> xen_make_pte
>
> if (unlikely(pte & _PAGE_IOMAP) &&
>        (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
>    pte = iomap_pte(pte);
> } else {
>    pte &= ~_PAGE_IOMAP;
>    pte = pte_pfn_to_mfn(pte);
> }
>
> considering that in this case xen_initial_domain() returns false and
> addr is < ISA_END_ADDRESS (it is a gpfn address), xen_make_pte is going
> to threat the mfn as a pfn erroneously.
>
> In your patch you replaced pte_val_ma with pte_val that does the
> opposite translation (mfn -> pfn) so the end result is that you get the
> original mfn in rmd->mmu_update->val.
>

Indeed!

> The real fix should something along these lines:
>
>
>
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 3dd53f9..f2fadfc 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
>                pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
>        }
>
> -       if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
> +       if (pteval & _PAGE_IOMAP)
>                return pteval;
>
>        return pte_mfn_to_pfn(pteval);
> @@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
>         * mappings are just dummy local mappings to keep other
>         * parts of the kernel happy.
>         */
> -       if (unlikely(pte & _PAGE_IOMAP) &&
> -           (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
> +       if (unlikely(pte & _PAGE_IOMAP)) {
>                pte = iomap_pte(pte);
>        } else {
>                pte &= ~_PAGE_IOMAP;
> ---
>
> Could you please confirm whether this patch fixes your problem?


Sorry, it did not succeed. The Linux stubdom kernel crashed during
booting. The debug info is as follows.


<5>Linux version 2.6.32.41 (root@localhost.localdomain) (gcc version
4.4.1 20090725 (Red Hat 4.4.1-2) (GCC) ) #1 Wed Nov 9 15:26:21 GMT
2011
<6>KERNEL supported cpus:
<6>  Intel GenuineIntel
<6>  AMD AuthenticAMD
<6>  NSC Geode by NSC
<6>  Cyrix CyrixInstead
<6>  Centaur CentaurHauls
<6>  Transmeta GenuineTMx86
<6>  Transmeta TransmetaCPU
<6>  UMC UMC UMC UMC
<6>released 0 pages of unused memory
<6>BIOS-provided physical RAM map:
<6> Xen: 0000000000000000 - 00000000000a0000 <c>(usable)<c>
<6> Xen: 00000000000a0000 - 0000000000100000 <c>(reserved)<c>
<6> Xen: 0000000000100000 - 0000000004000000 <c>(usable)<c>
(XEN) mm.c:859:d36 Non-privileged (36) attempt to map I/O space 000000f0
(XEN) mm.c:5046:d36 ptwr_emulate: could not get_page_from_l1e()
(XEN) d36:v0: unhandled page fault (ec=0003)
(XEN) Pagetable walk from c038c000:
(XEN)  L3[0x003] = 000000009791d001 000003c8
(XEN)  L2[0x001] = 0000000097d34067 000012f1
(XEN)  L1[0x18c] = 0000000093df9061 0000038c
(XEN) domain_crash_sync called from entry.S (ff1ddf7c)
(XEN) Domain 36 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.2-unstable  x86_32p  debug=y  Not tainted ]----
(XEN) CPU:    3
(XEN) EIP:    e019:[<c01048bc>]
(XEN) EFLAGS: 00000246   EM: 1   CONTEXT: pv guest
(XEN) eax: 00000000   ebx: c038c000   ecx: 00000000   edx: c0320000
(XEN) esi: 000f0463   edi: 00000000   ebp: c038c000   esp: c0321e8c
(XEN) cr0: 8005003b   cr4: 000426f4   cr3: 00a96200   cr2: c038c000
(XEN) ds: e021   es: e021   fs: 0000   gs: 0000   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c0321e8c:
(XEN)    00000003 c01048bc 0001e019 00010046 f5600000 000f0000 00000000 c03563c8
(XEN)    000f0000 00000000 000f0000 00000000 000f0000 00000000 c035677e 00000563
(XEN)    80000000 c0368f78 c0368f7e c0349728 c0214090 00000010 00000000 000001ff
(XEN)    00000000 c036b550 c038ed40 c0321fd4 c035683e 00000563 80000000 c0362757
(XEN)    00000000 c0368f7e 00000000 c02de171 00000000 00000000 c036b550 c038ed40
(XEN)    c0321fd4 00000000 c036b550 c038ed40 c0321fd4 c034cf01 00000004 00000000
(XEN)    ffffffff 0000000a ffffffff ffffffff c0321fc4 00000035 00000000 c0321fd0
(XEN)    ffffffff ffffffff c0393360 c0393160 00000200 c02e3791 00000004 00000000
(XEN)    ffffffff 0000000a ffffffff c0321fcc c0214090 00000090 c02de1cc c0321fcc
(XEN)    c029f093 c0321fd0 02040800 00534000 c032aa7c 00000000 c03494d5 c02de1cc
(XEN)    c02a3020 c02e3791 c036ace0 02040800 c034b11c 00000000 1f898175 8c080201
(XEN)    02040800 0001067a 00000000 c12eb000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 9791d001 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000


>
> Konrad, do you know if this could have any unintended consequences?
> I don't think it can be a problem security wise because Xen is going to
> do all the permission checks anyway.
> The only problem I can see is if a domU is going to call xen_make_pte
> with _PAGE_IOMAP and a pfn->mfn translation is supposed to happen.

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5321 bytes --]

On Wed, 9 Nov 2011, Jiageng Yu wrote:
> 2011/11/9 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> >
> > On Thu, 27 Oct 2011, Jiageng Yu wrote:
> > > Hi Stefano,
> > >
> > >       I have some progress in linux based stubdom project. As shown in the
> > > attached video, I have started the emulated vga device in the linux
> > > based stubdom.
> > >
> > >       In a short conclusion, for the linux based stubdom, there are two
> > > major problems about vga device emulation. The first is the vram
> > > mapping, which we discussed a lot previously and handled it.
> >
> > Do you have an updated version of the patch you used?
> >
> >
> > > Another
> > > is the vga BIOS mapping (address 0xc0000-0xc8fff of hvm guest).
> >
> > This is caused by qemu trying to map that memory area in its own address
> > space, right?
> >
> >
> > >       I found the vga BIOS mapping problem in remap_area_mfn_pte_fn()
> > > function. The pte_mkspecial() will return invalid value when I try to
> > > map 0xc0000-0xc8fff into linux based stubdom.
> >
> > What is exactly the error you are seeing?
> >
> >
> > > pte_mkspecial()
> > >       ->pte_set_flags()
> > >               ->native_pte_val()
> > >               ->native_make_pte()
> > >
> > >       According to my test, the root cause of vga BIOS mapping problem is
> > > native_xxx functions. We could avoid the problem by invoking functions
> > > defined in paravirt.h instead. The patch is as follows. But I think it
> > > is not a good way to handle the problem. Maybe you can give me some
> > > suggestions.
> > >
> > >       I also found the hard disk didnÃ??Ã?????Ã??Ã????Ã??Ã???Ã??Ã??Ã??Ã?¢??t work well. I will investigate it these days.
> > >
> > >
> > > --- a/arch/x86/xen/mmu.c
> > > +++ b/arch/x86/xen/mmu.c
> > > @@ -2639,12 +2640,16 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
> > > pgtable_t token,
> > >                                unsigned long addr, void *data)
> > >  {
> > >       struct remap_data *rmd = data;
> > > -     pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> > > +    if((rmd->mfn & 0xfffffff0) == 0xc0){
> > > +         pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
> > > +         rmd->mmu_update->val = pte_val(pte);
> > > +    }else{
> > > +         pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> > > +         rmd->mmu_update->val = pte_val_ma(pte);
> > > +    }
> >
> > Even if the fix is not the correct one I think I might understand what
> > the real problem is:
> >
> > pfn_pte -> xen_make_pte
> >
> > if (unlikely(pte & _PAGE_IOMAP) &&
> >        (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
> >    pte = iomap_pte(pte);
> > } else {
> >    pte &= ~_PAGE_IOMAP;
> >    pte = pte_pfn_to_mfn(pte);
> > }
> >
> > considering that in this case xen_initial_domain() returns false and
> > addr is < ISA_END_ADDRESS (it is a gpfn address), xen_make_pte is going
> > to threat the mfn as a pfn erroneously.
> >
> > In your patch you replaced pte_val_ma with pte_val that does the
> > opposite translation (mfn -> pfn) so the end result is that you get the
> > original mfn in rmd->mmu_update->val.
> >
> 
> Indeed!
> 
> > The real fix should something along these lines:
> >
> >
> >
> > diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> > index 3dd53f9..f2fadfc 100644
> > --- a/arch/x86/xen/mmu.c
> > +++ b/arch/x86/xen/mmu.c
> > @@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
> >                pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
> >        }
> >
> > -       if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
> > +       if (pteval & _PAGE_IOMAP)
> >                return pteval;
> >
> >        return pte_mfn_to_pfn(pteval);
> > @@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
> >         * mappings are just dummy local mappings to keep other
> >         * parts of the kernel happy.
> >         */
> > -       if (unlikely(pte & _PAGE_IOMAP) &&
> > -           (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
> > +       if (unlikely(pte & _PAGE_IOMAP)) {
> >                pte = iomap_pte(pte);
> >        } else {
> >                pte &= ~_PAGE_IOMAP;
> > ---
> >
> > Could you please confirm whether this patch fixes your problem?
> 
> 
> Sorry, it did not succeed. The Linux stubdom kernel crashed during
> booting. The debug info is as follows.

The kernel is tring to access some memory < ISA_END_ADDRESS that is
supposed to be translated (pfn->mfn), but it doesn't happen anymore
because of that patch.

At this point the only solution I can think of is changing
remap_area_mfn_pte_fn:


diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 3dd53f9..dd088f2 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2350,7 +2350,8 @@ static int remap_area_mfn_pte_fn(pte_t *ptep, pgtable_t token,
 				 unsigned long addr, void *data)
 {
 	struct remap_data *rmd = data;
-	pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
+	pte_t pte = pte_mkspecial(native_make_pte(((phys_addr_t)(rmd->mfn++)
+					<< PAGE_SHIFT) | massage_pgprot(rmd->prot)));
 
 	rmd->mmu_update->ptr = virt_to_machine(ptep).maddr;
 	rmd->mmu_update->val = pte_val_ma(pte);

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

2011/11/9 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Wed, 9 Nov 2011, Jiageng Yu wrote:
>> 2011/11/9 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>> >
>> > On Thu, 27 Oct 2011, Jiageng Yu wrote:
>> > > Hi Stefano,
>> > >
>> > >       I have some progress in linux based stubdom project. As shown in the
>> > > attached video, I have started the emulated vga device in the linux
>> > > based stubdom.
>> > >
>> > >       In a short conclusion, for the linux based stubdom, there are two
>> > > major problems about vga device emulation. The first is the vram
>> > > mapping, which we discussed a lot previously and handled it.
>> >
>> > Do you have an updated version of the patch you used?
>> >
>> >
>> > > Another
>> > > is the vga BIOS mapping (address 0xc0000-0xc8fff of hvm guest).
>> >
>> > This is caused by qemu trying to map that memory area in its own address
>> > space, right?
>> >
>> >
>> > >       I found the vga BIOS mapping problem in remap_area_mfn_pte_fn()
>> > > function. The pte_mkspecial() will return invalid value when I try to
>> > > map 0xc0000-0xc8fff into linux based stubdom.
>> >
>> > What is exactly the error you are seeing?
>> >
>> >
>> > > pte_mkspecial()
>> > >       ->pte_set_flags()
>> > >               ->native_pte_val()
>> > >               ->native_make_pte()
>> > >
>> > >       According to my test, the root cause of vga BIOS mapping problem is
>> > > native_xxx functions. We could avoid the problem by invoking functions
>> > > defined in paravirt.h instead. The patch is as follows. But I think it
>> > > is not a good way to handle the problem. Maybe you can give me some
>> > > suggestions.
>> > >
>> > >       I also found the hard disk didnÃ??Ã?????Ã??Ã????Ã??Ã???Ã??Ã??Ã??Ã?¢??t work well. I will investigate it these days.
>> > >
>> > >
>> > > --- a/arch/x86/xen/mmu.c
>> > > +++ b/arch/x86/xen/mmu.c
>> > > @@ -2639,12 +2640,16 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
>> > > pgtable_t token,
>> > >                                unsigned long addr, void *data)
>> > >  {
>> > >       struct remap_data *rmd = data;
>> > > -     pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
>> > > +    if((rmd->mfn & 0xfffffff0) == 0xc0){
>> > > +         pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
>> > > +         rmd->mmu_update->val = pte_val(pte);
>> > > +    }else{
>> > > +         pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
>> > > +         rmd->mmu_update->val = pte_val_ma(pte);
>> > > +    }
>> >
>> > Even if the fix is not the correct one I think I might understand what
>> > the real problem is:
>> >
>> > pfn_pte -> xen_make_pte
>> >
>> > if (unlikely(pte & _PAGE_IOMAP) &&
>> >        (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
>> >    pte = iomap_pte(pte);
>> > } else {
>> >    pte &= ~_PAGE_IOMAP;
>> >    pte = pte_pfn_to_mfn(pte);
>> > }
>> >
>> > considering that in this case xen_initial_domain() returns false and
>> > addr is < ISA_END_ADDRESS (it is a gpfn address), xen_make_pte is going
>> > to threat the mfn as a pfn erroneously.
>> >
>> > In your patch you replaced pte_val_ma with pte_val that does the
>> > opposite translation (mfn -> pfn) so the end result is that you get the
>> > original mfn in rmd->mmu_update->val.
>> >
>>
>> Indeed!
>>
>> > The real fix should something along these lines:
>> >
>> >
>> >
>> > diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
>> > index 3dd53f9..f2fadfc 100644
>> > --- a/arch/x86/xen/mmu.c
>> > +++ b/arch/x86/xen/mmu.c
>> > @@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
>> >                pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
>> >        }
>> >
>> > -       if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
>> > +       if (pteval & _PAGE_IOMAP)
>> >                return pteval;
>> >
>> >        return pte_mfn_to_pfn(pteval);
>> > @@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
>> >         * mappings are just dummy local mappings to keep other
>> >         * parts of the kernel happy.
>> >         */
>> > -       if (unlikely(pte & _PAGE_IOMAP) &&
>> > -           (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
>> > +       if (unlikely(pte & _PAGE_IOMAP)) {
>> >                pte = iomap_pte(pte);
>> >        } else {
>> >                pte &= ~_PAGE_IOMAP;
>> > ---
>> >
>> > Could you please confirm whether this patch fixes your problem?
>>
>>
>> Sorry, it did not succeed. The Linux stubdom kernel crashed during
>> booting. The debug info is as follows.
>
> The kernel is tring to access some memory < ISA_END_ADDRESS that is
> supposed to be translated (pfn->mfn), but it doesn't happen anymore
> because of that patch.
>
> At this point the only solution I can think of is changing
> remap_area_mfn_pte_fn:
>
>
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 3dd53f9..dd088f2 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -2350,7 +2350,8 @@ static int remap_area_mfn_pte_fn(pte_t *ptep, pgtable_t token,
>                                 unsigned long addr, void *data)
>  {
>        struct remap_data *rmd = data;
> -       pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
> +       pte_t pte = pte_mkspecial(native_make_pte(((phys_addr_t)(rmd->mfn++)
> +                                       << PAGE_SHIFT) | massage_pgprot(rmd->prot)));
>
>        rmd->mmu_update->ptr = virt_to_machine(ptep).maddr;
>        rmd->mmu_update->val = pte_val_ma(pte);


Great! It works.

The keyboard driver is OK now. I am working on network device. In
linux stubdom, I have udev, ifconfig and brctl tools. After udevd
started, stubdom executes "ifconfig eth0 IPadderss netmask netgate up"
to setup the network. When qemu in stubdom creates a tapxx interface
for hvm guest,  the script should be executed to build a net bridge.

      /sbin/brctl addbr eth0
      /sbin/brctl addif eth0 tapXX

Therefore, the hvm guest has the network device. Is this plan
reasonable? Or have better one?

Thanks.


Jiageng Yu.

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Wed, 9 Nov 2011, Jiageng Yu wrote:
> The keyboard driver is OK now. I am working on network device. In
> linux stubdom, I have udev, ifconfig and brctl tools. After udevd
> started, stubdom executes "ifconfig eth0 IPadderss netmask netgate up"
> to setup the network. When qemu in stubdom creates a tapxx interface
> for hvm guest,  the script should be executed to build a net bridge.
> 
>       /sbin/brctl addbr eth0
>       /sbin/brctl addif eth0 tapXX
> 
> Therefore, the hvm guest has the network device. Is this plan
> reasonable? Or have better one?

The bridge should be called xenbr0, the stubdom's network interface
(that should probably called eth0) should be added to the bridge at boot
time.

Like you said, when qemu starts is going to create a tap interface, on
Linux usually we rely on a udev script to add the tap interface to the
bridge. The script is tools/hotplug/Linux/vif-setup, that calls
tools/hotplug/Linux/vif-bridge.

So at the end you have:

xenbr0 (bridge)
||
|+-------------------------------+
|                                |
eth0                             |
(stubdom network interface)      tapXX
                                 (qemu's tap interface)

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1: Type: text/plain, Size: 3507 bytes --]

2011/11/10 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>
> On Wed, 9 Nov 2011, Jiageng Yu wrote:
> > The keyboard driver is OK now. I am working on network device. In
> > linux stubdom, I have udev, ifconfig and brctl tools. After udevd
> > started, stubdom executes "ifconfig eth0 IPadderss netmask netgate up"
> > to setup the network. When qemu in stubdom creates a tapxx interface
> > for hvm guest,  the script should be executed to build a net bridge.
> >
> >       /sbin/brctl addbr eth0
> >       /sbin/brctl addif eth0 tapXX
> >
> > Therefore, the hvm guest has the network device. Is this plan
> > reasonable? Or have better one?
>
> The bridge should be called xenbr0, the stubdom's network interface
> (that should probably called eth0) should be added to the bridge at boot
> time.
>
> Like you said, when qemu starts is going to create a tap interface, on
> Linux usually we rely on a udev script to add the tap interface to the
> bridge. The script is tools/hotplug/Linux/vif-setup, that calls
> tools/hotplug/Linux/vif-bridge.
>
> So at the end you have:
>
> xenbr0 (bridge)
> ||
> |+-------------------------------+
> |                                |
> eth0                             |
> (stubdom network interface)      tapXX
>                                 (qemu's tap interface)


Hi Stefano,

    I have a prototype of network of linux based stubdom, as shown in
attached figure. I list my design points, please comment on them.

    1. Qemu-ifup script in Linux stubdom.

     Qemu in stubdom invokes qemu-ifup script to setup the
bridge(net/tap.c). Because the linux stubdom only has nash and can not
execute the qemu-ifup script, I implement a c version of qemu-ifup
script. Using the following way, the qemu will invoke qemu-ifup
program in stubdom.

diff -r 0f36c2eec2e1 tools/libxl/libxl_dm.c
--- a/tools/libxl/libxl_dm.c	Thu Jul 28 15:40:54 2011 +0100
+++ b/tools/libxl/libxl_dm.c	Thu Nov 17 22:41:29 2011 +0000
@@ -29,9 +29,12 @@
 #include "libxl.h"
 #include "flexarray.h"

-static const char *libxl_tapif_script(libxl__gc *gc)
+static const char *libxl_tapif_script(libxl__gc *gc,
+                                      libxl_device_model_info *info)
 {
 #ifdef __linux__
+    if(info->device_model_linux_stubdomain)
+        return libxl__sprintf(gc, "/bin/qemu-ifup");
     return libxl__strdup(gc, "no");
 #else
     return libxl__sprintf(gc, "%s/qemu-ifup", libxl_xen_script_dir_path());

     I do not use libxl_xen_script_dir_path() to determine the path of
qemu-ifup, because we don't want include xen-unstable.hg/Config.mk in
linux stubdom. Therefore, I hardcoded this path.

    2. Network tools.

    Our linux based stubdom do not have the real shell and IP stack,
so we must custom the network tools. I notice the bridge-utils-1.5
version creates AF_LOCAL socket, so brctl can be used without
modification. But ifconfig would not be so luck. I need to rewrite
ifconfig and make it only support bring up the interfaces.

    3. The mac address.

    If we declare the mac address in stubdom-cfg file, the eth0 in
stubdom and eth0 in hvm guest will be set to the same mac address.

   do_domain_create (or libxl__create_stubdom)
         -->libxl_device_nic_add

   As a temporary solution, I hardcoded a static mac address for linux
stdubom in libxl__create_stubdom().


  Thanks.


Jiageng Yu.

[-- Attachment #2: Screenshot-QEMU (fedora14-dm)-1.png --]
[-- Type: image/png, Size: 21441 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4206 bytes --]

On Thu, 17 Nov 2011, Jiageng Yu wrote:
> 2011/11/10 Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> >
> > On Wed, 9 Nov 2011, Jiageng Yu wrote:
> > > The keyboard driver is OK now. I am working on network device. In
> > > linux stubdom, I have udev, ifconfig and brctl tools. After udevd
> > > started, stubdom executes "ifconfig eth0 IPadderss netmask netgate up"
> > > to setup the network. When qemu in stubdom creates a tapxx interface
> > > for hvm guest, � the script should be executed to build a net bridge.
> > >
> > > �  �  �  /sbin/brctl addbr eth0
> > > �  �  �  /sbin/brctl addif eth0 tapXX
> > >
> > > Therefore, the hvm guest has the network device. Is this plan
> > > reasonable? Or have better one?
> >
> > The bridge should be called xenbr0, the stubdom's network interface
> > (that should probably called eth0) should be added to the bridge at boot
> > time.
> >
> > Like you said, when qemu starts is going to create a tap interface, on
> > Linux usually we rely on a udev script to add the tap interface to the
> > bridge. The script is tools/hotplug/Linux/vif-setup, that calls
> > tools/hotplug/Linux/vif-bridge.
> >
> > So at the end you have:
> >
> > xenbr0 (bridge)
> > ||
> > |+-------------------------------+
> > | �  �  �  �  �  �  �  �  �  �  �  �  �  �  �  � |
> > eth0 �  �  �  �  �  �  �  �  �  �  �  �  �  �  |
> > (stubdom network interface) �  �  � tapXX
> > �  �  �  �  �  �  �  �  �  �  �  �  �  �  �  �  (qemu's tap interface)
> 
> 
> Hi Stefano,
> 
> �  � � I have a prototype of network of linux based stubdom, as shown in
> attached figure. I list my design� points, please comment on them.
> 
>     1. Qemu-ifup script in Linux stubdom.
> 
>      Qemu in stubdom invokes qemu-ifup script to setup the
> bridge(net/tap.c). Because the linux stubdom only has nash and can not
> execute the qemu-ifup script, I implement a c version of qemu-ifup
> script. Using the following way, the qemu will invoke qemu-ifup
> program in stubdom.
> 
> diff -r 0f36c2eec2e1 tools/libxl/libxl_dm.c
> --- a/tools/libxl/libxl_dm.c	Thu Jul 28 15:40:54 2011 +0100
> +++ b/tools/libxl/libxl_dm.c	Thu Nov 17 22:41:29 2011 +0000
> @@ -29,9 +29,12 @@
>  #include "libxl.h"
>  #include "flexarray.h"
> 
> -static const char *libxl_tapif_script(libxl__gc *gc)
> +static const char *libxl_tapif_script(libxl__gc *gc,
> +                                      libxl_device_model_info *info)
>  {
>  #ifdef __linux__
> +    if(info->device_model_linux_stubdomain)
> +        return libxl__sprintf(gc, "/bin/qemu-ifup");
>      return libxl__strdup(gc, "no");
>  #else
>      return libxl__sprintf(gc, "%s/qemu-ifup", libxl_xen_script_dir_path());
> 
>      I do not use libxl_xen_script_dir_path() to determine the path of
> qemu-ifup, because we don't want include xen-unstable.hg/Config.mk in
> linux stubdom. Therefore, I hardcoded this path.

That is OK.


>     2. Network tools.
> 
>     Our linux based stubdom do not have the real shell and IP stack,
> so we must custom the network tools. I notice the bridge-utils-1.5
> version creates AF_LOCAL socket, so brctl can be used without
> modification. But ifconfig would not be so luck. I need to rewrite
> ifconfig and make it only support bring up the interfaces.

I think you need to call a couple of ioctl in order to enable a
network interface.


>     3. The mac address.
> 
>     If we declare the mac address in stubdom-cfg file, the eth0 in
> stubdom and eth0 in hvm guest will be set to the same mac address.
> 
>    do_domain_create (or libxl__create_stubdom)
>          -->libxl_device_nic_add
> 
>    As a temporary solution, I hardcoded a static mac address for linux
> stdubom in libxl__create_stubdom().

The mac address of eth0 (and xenbr0) in the stubdom is not
important considering that is never going to be used: all the traffic
should go through the tap interface anyway.
It could even be fe:ff:ff:ff:ff:ff.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Linux Stubdom Problem

[Konrad Rzeszutek Wilk]

> The real fix should something along these lines:
> 
> 
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 3dd53f9..f2fadfc 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
>  		pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
>  	}
>  
> -	if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
> +	if (pteval & _PAGE_IOMAP)
>  		return pteval;
>  
>  	return pte_mfn_to_pfn(pteval);
> @@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
>  	 * mappings are just dummy local mappings to keep other
>  	 * parts of the kernel happy.
>  	 */
> -	if (unlikely(pte & _PAGE_IOMAP) &&
> -	    (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
> +	if (unlikely(pte & _PAGE_IOMAP)) {
>  		pte = iomap_pte(pte);
>  	} else {
>  		pte &= ~_PAGE_IOMAP;
> ---
> 
> Could you please confirm whether this patch fixes your problem?
> 
> Konrad, do you know if this could have any unintended consequences?
> I don't think it can be a problem security wise because Xen is going to
> do all the permission checks anyway.
> The only problem I can see is if a domU is going to call xen_make_pte
> with _PAGE_IOMAP and a pfn->mfn translation is supposed to happen.

I am not sure. I think I tried this at some point but ran in some problems, but
I can't recall which ones. Let me stick this in my #testing branch and see how it
fares with Dom0/DomU/DomU PCI/ etc.

Re: Re: Linux Stubdom Problem

[Stefano Stabellini]

On Wed, 9 Nov 2011, Konrad Rzeszutek Wilk wrote:
> > The real fix should something along these lines:
> > 
> > 
> > 
> > diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> > index 3dd53f9..f2fadfc 100644
> > --- a/arch/x86/xen/mmu.c
> > +++ b/arch/x86/xen/mmu.c
> > @@ -422,7 +422,7 @@ static pteval_t xen_pte_val(pte_t pte)
> >  		pteval = (pteval & ~_PAGE_PAT) | _PAGE_PWT;
> >  	}
> >  
> > -	if (xen_initial_domain() && (pteval & _PAGE_IOMAP))
> > +	if (pteval & _PAGE_IOMAP)
> >  		return pteval;
> >  
> >  	return pte_mfn_to_pfn(pteval);
> > @@ -483,8 +483,7 @@ static pte_t xen_make_pte(pteval_t pte)
> >  	 * mappings are just dummy local mappings to keep other
> >  	 * parts of the kernel happy.
> >  	 */
> > -	if (unlikely(pte & _PAGE_IOMAP) &&
> > -	    (xen_initial_domain() || addr >= ISA_END_ADDRESS)) {
> > +	if (unlikely(pte & _PAGE_IOMAP)) {
> >  		pte = iomap_pte(pte);
> >  	} else {
> >  		pte &= ~_PAGE_IOMAP;
> > ---
> > 
> > Could you please confirm whether this patch fixes your problem?
> > 
> > Konrad, do you know if this could have any unintended consequences?
> > I don't think it can be a problem security wise because Xen is going to
> > do all the permission checks anyway.
> > The only problem I can see is if a domU is going to call xen_make_pte
> > with _PAGE_IOMAP and a pfn->mfn translation is supposed to happen.
> 
> I am not sure. I think I tried this at some point but ran in some problems, but
> I can't recall which ones. Let me stick this in my #testing branch and see how it
> fares with Dom0/DomU/DomU PCI/ etc.


Actually the patch above caused troubles at boot time because when the
guest tries to map memory below ISA_END_ADDRESS pfns don't get converted
to mfns anymore.
I ended up suggesting to modify remap_area_mfn_pte_fn instead, see patch
below.

---

Use native_make_pte in remap_area_mfn_pte_fn because
remap_area_mfn_pte_fn gets always called with _PAGE_IOMAP set so no
pfn->mfn translations are necessary. At the same time
xen_initial_domain() returns false in stubdoms so if we are trying to
remap memory < ISA_END_ADDRESS from another guest xen_make_pte would
force a pfn->mfn translation.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 3dd53f9..f628441 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2350,10 +2350,19 @@ static int remap_area_mfn_pte_fn(pte_t *ptep, pgtable_t token,
 				 unsigned long addr, void *data)
 {
 	struct remap_data *rmd = data;
-	pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
-
+	/* Use the native_make_pte function because we are sure we don't
+	 * have to do any pfn->mfn translations but at the same time we
+	 * could in a stubdom so xen_initial_domain() would return false. */
+	pte_t pte = pte_mkspecial(native_make_pte(((phys_addr_t)(rmd->mfn++)
+					<< PAGE_SHIFT) | massage_pgprot(rmd->prot)));
+	pteval_t val = pte_val_ma(pte);
+
+	if (pat_enabled && !WARN_ON(val & _PAGE_PAT)) {
+		if ((val & (_PAGE_PCD | _PAGE_PWT)) == _PAGE_PWT)
+			val = (val & ~(_PAGE_PCD | _PAGE_PWT)) | _PAGE_PAT;
+	}
 	rmd->mmu_update->ptr = virt_to_machine(ptep).maddr;
-	rmd->mmu_update->val = pte_val_ma(pte);
+	rmd->mmu_update->val = val;
 	rmd->mmu_update++;
 
 	return 0;

Re: Re: Linux Stubdom Problem

[Jiageng Yu]

[-- Attachment #1: Type: text/plain, Size: 8717 bytes --]

2011/9/15 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:
> On Wed, 14 Sep 2011, Jiageng Yu wrote:
>> Hi Stefano,
>>
>>      I just have a prototype of vram mapping and test it now. The
>> implementation of linux-stubdom kernel part is as follows.
>> xen_remap_domain_mfn_range2 function maps foreign dom's physical
>> address into linux kernel space. It is similar to
>> xen_remap_domain_mfn_range. But xen_remap_domain_mfn_range is used to
>> map foreign pages into linux user space.
>>
>>     But the page info seems wrong after executing xen_remap_domain_mfn_range2.
>>
>>     struct page *page=pfn_to_page(vmalloc_to_pfn(info->fb));
>>
>>     The page->_count = 0xc2c2c2c2. It is very strange.
>>
>>     Did I do the right thing?
>>
>
> use page_address instead of pfn_to_page to find the struct page
>
>
>>     Greeting.
>>
>> Jiageng Yu.
>>
>>
>> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
>> index 204e3ba..72a7808 100644
>> --- a/arch/x86/xen/mmu.c
>> +++ b/arch/x86/xen/mmu.c
>> @@ -2693,6 +2693,73 @@ out:
>>  }
>>  EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
>>
>> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long gpfn,
>> +                int nr, unsigned domid)
>> +{
>> +    struct remap_data rmd;
>> +    struct mmu_update mmu_update[REMAP_BATCH_SIZE];
>> +    int level,i,batch,nr_page = nr;
>> +    unsigned long range;
>> +    int err = 0;
>> +    unsigned long vaddr,base_addr = addr;
>> +    pte_t pte,*ptep;
>> +
>> +    rmd.mfn = gpfn;
>> +    rmd.prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |
>> _PAGE_IOMAP);
>> +
>> +    while(nr_page) {
>> +        batch = min(REMAP_BATCH_SIZE, nr);
>> +        range = (unsigned long)batch << PAGE_SHIFT;
>> +
>> +        rmd.mmu_update = mmu_update;
>> +
>> +        for(i=0; i < batch; i++){
>> +            pte = pte_mkspecial(pfn_pte(rmd.mfn++, rmd.prot));
>> +            vaddr = base_addr + i*PAGE_SIZE;
>> +            ptep = lookup_address(vaddr, &level);
>
> you need to check if ptep is valid here and the level is PG_LEVEL_4K
>
>> +            rmd.mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr |
>> +                                        MMU_NORMAL_PT_UPDATE;
>
> you can use pte_mfn(*ptep) instead of arbitrary_virt_to_machine
>
>
>> +            rmd.mmu_update->val = pte_val_ma(pte);
>> +            rmd.mmu_update++;
>> +        }
>> +
>> +        err = -EFAULT;
>> +        if(HYPERVISOR_mmu_update(mmu_update, batch, NULL, domid) < 0)
>> +            goto out;
>> +
>> +        nr_page -= batch;
>> +        base_addr += range;
>> +    }
>> +
>> +    err = 0;
>> +
>> +    base_addr = addr;
>> +    for(i=0; i < nr; i++){
>> +        vaddr = base_addr + i*PAGE_SIZE;
>> +        set_phys_to_machine(vmalloc_to_pfn(vaddr),
>> +                arbitrary_virt_to_machine(vaddr).maddr >> PAGE_SHIFT);
>> +    }
>
> The second argument (mfn) to set_phys_to_machine is wrong:
> arbitrary_virt_to_machine ends up calling virt_to_machine if
> virt_addr_valid. You need to manually call pte_mfn:
>
> /* the ptep content has been updated by Xen so we can lookup the foreign
>  * mfn from the pte now */
> pte = lookup_address(vaddr, &level);
> BUG_ON(pte == NULL);
> offset = vaddr & ~PAGE_MASK;
> mfn = XMADDR(((phys_addr_t)pte_mfn(*pte) << PAGE_SHIFT) + offset);
>
>
>> +
>> +out:
>> +     flush_tlb_all();
>> +     return err;
>> +}
>> +EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range2);
>
> the name should be changed to xen_remap_foreign_gpfn_range
>
>
>>  #ifdef CONFIG_XEN_PVHVM
>>  static void xen_hvm_exit_mmap(struct mm_struct *mm)
>>  {
>> diff --git a/drivers/video/xen-fbfront.c b/drivers/video/xen-fbfront.c
>> index dc72563..82da2ee 100644
>> --- a/drivers/video/xen-fbfront.c
>> +++ b/drivers/video/xen-fbfront.c
>> @@ -25,8 +25,12 @@
>>  #include <linux/module.h>
>>  #include <linux/vmalloc.h>
>>  #include <linux/mm.h>
>> +#include <linux/sched.h>
>> +#include <asm/pgtable.h>
>> +#include <asm/page.h>
>>
>>  #include <asm/xen/hypervisor.h>
>> +#include <asm/xen/page.h>
>>
>>  #include <xen/xen.h>
>>  #include <xen/events.h>
>> @@ -34,6 +38,7 @@
>>  #include <xen/interface/io/fbif.h>
>>  #include <xen/interface/io/protocols.h>
>>  #include <xen/xenbus.h>
>> +#include <xen/xen-ops.h>
>>
>>  struct xenfb_info {
>>       unsigned char           *fb;
>> @@ -62,6 +67,12 @@ module_param_array(video, int, NULL, 0);
>>  MODULE_PARM_DESC(video,
>>       "Video memory size in MB, width, height in pixels (default 2,800,600)");
>>
>> +static unsigned long foreign_vaddr = 0;
>> +module_param(foreign_vaddr, ulong, S_IRUGO);
>> +
>> +static unsigned long foreign_domid = 0;
>> +module_param(foreign_domid, ulong, S_IRUGO);
>> +
>>  static void xenfb_make_preferred_console(void);
>>  static int xenfb_remove(struct xenbus_device *);
>>  static void xenfb_init_shared_page(struct xenfb_info *, struct fb_info *);
>> @@ -398,7 +408,17 @@ static int __devinit xenfb_probe(struct xenbus_device *dev,
>>       if (info->fb == NULL)
>>               goto error_nomem;
>>       memset(info->fb, 0, fb_size);
>> -
>> +    if((foreign_vaddr != 0) && (foreign_domid != 0)){
>> +        ret = xen_remap_domain_mfn_range2((unsigned long)(info->fb),
>> +                                    foreign_vaddr >> PAGE_SHIFT,
>> +                                    fb_size >> PAGE_SHIFT, foreign_domid);
>
> you should rename foreign_vaddr to foreign_gpfn and pass the gpfn value
> that is the ram_addr (page shifted) passed to xen_ram_alloc in qemu.
>
>> +        if(ret < 0){
>> +            printk("Can not remap vram of hvm guest.\n");
>> +            goto error;
>> +        }
>> +    }
>>       info->nr_pages = (fb_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
>>
>>       info->mfns = vmalloc(sizeof(unsigned long) * info->nr_pages);
>> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
>> index 4349e89..1554531 100644
>> --- a/include/xen/xen-ops.h
>> +++ b/include/xen/xen-ops.h
>> @@ -20,6 +20,10 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
>>                              unsigned long mfn, int nr,
>>                              pgprot_t prot, unsigned domid);
>>
>> +int xen_remap_domain_mfn_range2(unsigned long addr,unsigned long mfn,
>> +                int nr, unsigned domid);
>>  extern unsigned long *xen_contiguous_bitmap;
>>  int xen_create_contiguous_region(unsigned long vstart, unsigned int order,
>>                               unsigned int address_bits);
>>
>

Hi Stefano,

   Good news! We have the basic Linux based stubdom now, as shown in
attached figures. It can run, but the keyboard and network drivers
still need to be tested.

   In my recent work, I found the pte_mkspecial() (in linux stubdom
kernel) returned invalid value when I tried to map some memory spaces
into linux based stubdom from hvm guest.

pte_mkspecial()
       ->pte_set_flags()
               ->native_pte_val()
               ->native_make_pte()

  According to my test, the root cause of the problem is native_xxx
functions. The patch is as follows. But I think hardcoding is not a
good way to handle the problem. Maybe you can give me some
suggestions.

  Best Regards!

Jiageng Yu.


diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 204e3ba..c995139 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2639,12 +2640,23 @@ static int remap_area_mfn_pte_fn(pte_t *ptep,
pgtable_t token,
 				 unsigned long addr, void *data)
 {
 	struct remap_data *rmd = data;
-	pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
+    if(((rmd->mfn & 0xfffffff0) == 0xc0) ||
+            ((rmd->mfn & 0xfffffff0) == 0x90) ||
+                ((rmd->mfn & 0xfffffff0) == 0x00) ||
+                    ((rmd->mfn & 0xfffffff0) == 0x70) ||
+                        ((rmd->mfn & 0xfffffff0) == 0x10)){
+	    pte_t pte = pfn_pte(rmd->mfn++, rmd->prot);
+	    rmd->mmu_update->val = pte_val(pte);
+    }else{
+	    pte_t pte = pte_mkspecial(pfn_pte(rmd->mfn++, rmd->prot));
+	    rmd->mmu_update->val = pte_val_ma(pte);
+    }

 	rmd->mmu_update->ptr = arbitrary_virt_to_machine(ptep).maddr;
-	rmd->mmu_update->val = pte_val_ma(pte);
 	rmd->mmu_update++;
-
 	return 0;
 }

[-- Attachment #2: Screenshot-QEMU (fedora14-dm)-1.png --]
[-- Type: image/png, Size: 9919 bytes --]

[-- Attachment #3: Screenshot-QEMU (fedora14-dm)-2.png --]
[-- Type: image/png, Size: 171106 bytes --]

[-- Attachment #4: Screenshot-QEMU (fedora14-dm)-3.png --]
[-- Type: image/png, Size: 9884 bytes --]

[-- Attachment #5: Screenshot-root@localhost_~-char-hvm.png --]
[-- Type: image/png, Size: 388741 bytes --]

[-- Attachment #6: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel