[Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Chris Lalancette]

Hello,
     oVirt is currently using straight x86_64 qemu emulation for certain parts
of the architecture (we mostly use KVM, but need to use full emulation for a
couple of parts).  We recently upgraded our userspace package to kvm-72, but
found that we could not PXE boot guests when we were doing full emulation (under
kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
similar results.  We ended up doing a bisect, and tracked down the problem to
this commit (from the kvm repo, but pulled from qemu):

http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda

The important changes in there in terms of this bug revolves around
TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what it was
before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in phys_page_find_alloc(), but
leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't claim
to understand the conditional code I've compiled out, so I'm not sure where the
bug would be.  Does anyone have an idea what the problem might be?

Thanks,
Chris Lalancette

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Anthony Liguori]

Chris Lalancette wrote:
> Hello,
>      oVirt is currently using straight x86_64 qemu emulation for certain parts
> of the architecture (we mostly use KVM, but need to use full emulation for a
> couple of parts).  We recently upgraded our userspace package to kvm-72, but
> found that we could not PXE boot guests when we were doing full emulation (under
> kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
> similar results.  We ended up doing a bisect, and tracked down the problem to
> this commit (from the kvm repo, but pulled from qemu):
>
> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda
>
> The important changes in there in terms of this bug revolves around
> TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what it was
> before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
> TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in phys_page_find_alloc(), but
> leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't claim
> to understand the conditional code I've compiled out, so I'm not sure where the
> bug would be.  Does anyone have an idea what the problem might be?
>   

Right now, the code just can't handle TARGET_PHYS_ADDR_SPACES_BITS > 
32.  This may help you:

diff --git a/exec.c b/exec.c
index a0aa7dc..cac7a87 100644
--- a/exec.c
+++ b/exec.c
@@ -153,7 +153,7 @@ typedef struct PhysPageDesc {
  */
 #define L1_BITS (TARGET_VIRT_ADDR_SPACE_BITS - L2_BITS - TARGET_PAGE_BITS)
 #else
-#define L1_BITS (32 - L2_BITS - TARGET_PAGE_BITS)
+#define L1_BITS (TARGET_PHYS_ADDR_SPACE_BITS - L2_BITS - TARGET_PAGE_BITS)
 #endif
 
 #define L1_SIZE (1 << L1_BITS)

But there are a lot more places in the code that assume 
TARGET_PHYS_ADDR_SPACE_BITS == 32.  I'm inclined to think that we should 
change it back to 32.

Regards,

Anthony Liguori

> Thanks,
> Chris Lalancette
>
>
>   

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Blue Swirl]

On 8/19/08, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Chris Lalancette wrote:
>
> > Hello,
> >     oVirt is currently using straight x86_64 qemu emulation for certain
> parts
> > of the architecture (we mostly use KVM, but need to use full emulation for
> a
> > couple of parts).  We recently upgraded our userspace package to kvm-72,
> but
> > found that we could not PXE boot guests when we were doing full emulation
> (under
> > kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
> > similar results.  We ended up doing a bisect, and tracked down the problem
> to
> > this commit (from the kvm repo, but pulled from qemu):
> >
> >
> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda
> >
> > The important changes in there in terms of this bug revolves around
> > TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what
> it was
> > before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
> > TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in
> phys_page_find_alloc(), but
> > leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't
> claim
> > to understand the conditional code I've compiled out, so I'm not sure
> where the
> > bug would be.  Does anyone have an idea what the problem might be?
> >
> >
>
>  Right now, the code just can't handle TARGET_PHYS_ADDR_SPACES_BITS > 32.
> This may help you:
>
>  diff --git a/exec.c b/exec.c
>  index a0aa7dc..cac7a87 100644
>  --- a/exec.c
>  +++ b/exec.c
>  @@ -153,7 +153,7 @@ typedef struct PhysPageDesc {
>   */
>  #define L1_BITS (TARGET_VIRT_ADDR_SPACE_BITS - L2_BITS - TARGET_PAGE_BITS)
>  #else
>  -#define L1_BITS (32 - L2_BITS - TARGET_PAGE_BITS)
>  +#define L1_BITS (TARGET_PHYS_ADDR_SPACE_BITS - L2_BITS - TARGET_PAGE_BITS)
>  #endif
>
>  #define L1_SIZE (1 << L1_BITS)
>
>  But there are a lot more places in the code that assume
> TARGET_PHYS_ADDR_SPACE_BITS == 32.  I'm inclined to think that we should
> change it back to 32.

Sparc32 uses 36 bits successfully, on SS-10/20/600MP a lot of devices
are located near the top of physical address space.

Sparc64 boots using address 0x1ff f000 0020 and that uses 42 bits (top
bit zero). I think I  have found a bug...

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Aurelien Jarno]

On Tue, Aug 19, 2008 at 07:46:50PM +0200, Chris Lalancette wrote:
> Hello,
>      oVirt is currently using straight x86_64 qemu emulation for certain parts
> of the architecture (we mostly use KVM, but need to use full emulation for a
> couple of parts).  We recently upgraded our userspace package to kvm-72, but
> found that we could not PXE boot guests when we were doing full emulation (under
> kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
> similar results.  We ended up doing a bisect, and tracked down the problem to
> this commit (from the kvm repo, but pulled from qemu):
> 
> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda
> 
> The important changes in there in terms of this bug revolves around
> TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what it was
> before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
> TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in phys_page_find_alloc(), but
> leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't claim
> to understand the conditional code I've compiled out, so I'm not sure where the
> bug would be.  Does anyone have an idea what the problem might be?
> 

Are you using qemu or qemu-system-x86_64? Could you also build qemu with
--disable-kqemu? It possible that kqemu support is causing this problem,
as it is limited to 32 bits.

I have tested support for PAE with various guests, including memtest86, 
and haven't encountered any problem.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32@debian.org         | aurelien@aurel32.net
   `-    people.debian.org/~aurel32 | www.aurel32.net

Re: Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Alan Pevec]

Anthony Liguori wrote:
>Chris Lalancette wrote:
>> Hello,
>>      oVirt is currently using straight x86_64 qemu emulation for 
>> certain parts
>> of the architecture (we mostly use KVM, but need to use full emulation 
>> for a
>> couple of parts).  We recently upgraded our userspace package to 
>> kvm-72, but
>> found that we could not PXE boot guests when we were doing full 
>> emulation (under
>> kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, 
>> with
>> similar results.  We ended up doing a bisect, and tracked down the 
>> problem to
>> this commit (from the kvm repo, but pulled from qemu):
>>
>> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda 
>>
>>
>> The important changes in there in terms of this bug revolves around
>> TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 
>> (what it was
>> before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
>> TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in 
>> phys_page_find_alloc(), but
>> leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I 
>> can't claim
>> to understand the conditional code I've compiled out, so I'm not sure 
>> where the
>> bug would be.  Does anyone have an idea what the problem might be?
>>   
> 
> Right now, the code just can't handle TARGET_PHYS_ADDR_SPACES_BITS > 
> 32.  This may help you:

I applied that to qemu svn trunk but still get the same tripple fault w/ pxelinux (syslinux-3.61-2.fc9)
qemu: fatal: triple fault
EAX=0f8ef0a8 EBX=f0711ae0 ECX=f07b0244 EDX=f07b026e
ESI=f0711adc EDI=00000001 EBP=0f8ee990 ESP=f07b0016
EIP=00000717 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 0f8ee990 ffffffff 0fcf938e
CS =0008 0f8ee990 ffffffff 0fcf9f8e
SS =0010 0f8ee990 ffffffff 0fcf938e
DS =0010 0f8ee990 ffffffff 0fcf938e
FS =0010 0f8ee990 ffffffff 0fcf938e
GS =0010 0f8ee990 ffffffff 0fcf938e
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     0f8f6ed0 00000037
IDT=     00000000 000003ff
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
CCS=0f8ee990 CCD=0f8efa5c CCO=ADDL    
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Chris Lalancette]

Aurelien Jarno wrote:
> On Tue, Aug 19, 2008 at 07:46:50PM +0200, Chris Lalancette wrote:
>> Hello,
>>      oVirt is currently using straight x86_64 qemu emulation for certain parts
>> of the architecture (we mostly use KVM, but need to use full emulation for a
>> couple of parts).  We recently upgraded our userspace package to kvm-72, but
>> found that we could not PXE boot guests when we were doing full emulation (under
>> kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
>> similar results.  We ended up doing a bisect, and tracked down the problem to
>> this commit (from the kvm repo, but pulled from qemu):
>>
>> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda
>>
>> The important changes in there in terms of this bug revolves around
>> TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what it was
>> before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
>> TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in phys_page_find_alloc(), but
>> leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't claim
>> to understand the conditional code I've compiled out, so I'm not sure where the
>> bug would be.  Does anyone have an idea what the problem might be?
>>

Sorry for the delay in responding.

> 
> Are you using qemu or qemu-system-x86_64? Could you also build qemu with
> --disable-kqemu? It possible that kqemu support is causing this problem,
> as it is limited to 32 bits.

I'm not sure what the difference between qemu and qemu-system-x86_64 is.  Can
you explain?  I've been testing with qemu-system-x86_64, for what it's worth.

I've now tried building qemu like this:

./configure --target-list=x86_64-softmmu,x86_64-linux-user && make -j8

When built like this, I get no problems PXE booting.  However, this is because
of this line in exec.c:

#elif defined(TARGET_X86_64) && !defined(USE_KQEMU)
#define TARGET_PHYS_ADDR_SPACE_BITS 42

So, in this case, we end up building with TARGET_PHYS_ADDR_SPACE_BITS == 32, so
things work.  However, if I build qemu like this:

./configure --target-list=x86_64-softmmu,x86_64-linux-user --disable-kqemu &&
make -j8

Then we fail on the PXE boot, because TARGET_PHYS_ADDR_BITS gets set to 42 in
that case.

Chris Lalancette

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Anthony Liguori]

Chris Lalancette wrote:
> Aurelien Jarno wrote:
>   
>> On Tue, Aug 19, 2008 at 07:46:50PM +0200, Chris Lalancette wrote:
>>     
>>> Hello,
>>>      oVirt is currently using straight x86_64 qemu emulation for certain parts
>>> of the architecture (we mostly use KVM, but need to use full emulation for a
>>> couple of parts).  We recently upgraded our userspace package to kvm-72, but
>>> found that we could not PXE boot guests when we were doing full emulation (under
>>> kvm, we could PXE boot just fine).  We also tried using qemu SVN tip, with
>>> similar results.  We ended up doing a bisect, and tracked down the problem to
>>> this commit (from the kvm repo, but pulled from qemu):
>>>
>>> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=commit;h=468f7507339a5236bff8ab339eb0c1b019a95fda
>>>
>>> The important changes in there in terms of this bug revolves around
>>> TARGET_PHYS_ADDR_SPACE_BITS in exec.c.  If I change that back to 32 (what it was
>>> before this patch for x86_64), the PXE boot succeeds.  Also, if I remove
>>> TARGET_PHYS_ADDR_SPACE_BITS > 32 conditional code in phys_page_find_alloc(), but
>>> leave TARGET_PHYS_ADDR_SPACE_BITS as 42, the PXE boot also works.  I can't claim
>>> to understand the conditional code I've compiled out, so I'm not sure where the
>>> bug would be.  Does anyone have an idea what the problem might be?
>>>
>>>       
>
> Sorry for the delay in responding.
>
>   
>> Are you using qemu or qemu-system-x86_64? Could you also build qemu with
>> --disable-kqemu? It possible that kqemu support is causing this problem,
>> as it is limited to 32 bits.
>>     
>
> I'm not sure what the difference between qemu and qemu-system-x86_64 is.  Can
> you explain?  I've been testing with qemu-system-x86_64, for what it's worth.
>   

qemu is really qemu-system-i386 just unfortunately named.  This define 
will differ in the qemu vs qemu-system-x86_64 binaries.

Has there been any resolution here?

Regards,

Anthony Liguori

Re: [Qemu-devel] Bug with TARGET_PHYS_ADDR_SPACE_BITS

[Chris Lalancette]

Anthony Liguori wrote:
> qemu is really qemu-system-i386 just unfortunately named.  This define 
> will differ in the qemu vs qemu-system-x86_64 binaries.
> 
> Has there been any resolution here?

Yes, see Glauber's post:

http://lists.gnu.org/archive/html/qemu-devel/2008-09/msg00076.html

and the resulting discussion.

Chris Lalancette