From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=41437 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OsehI-0001iy-L9 for qemu-devel@nongnu.org; Mon, 06 Sep 2010 12:37:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OsehE-00014V-0f for qemu-devel@nongnu.org; Mon, 06 Sep 2010 12:37:32 -0400 Received: from mail-vw0-f45.google.com ([209.85.212.45]:45863) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OsehD-000142-R7 for qemu-devel@nongnu.org; Mon, 06 Sep 2010 12:37:27 -0400 Received: by vws19 with SMTP id 19so3561331vws.4 for ; Mon, 06 Sep 2010 09:37:25 -0700 (PDT) MIME-Version: 1.0 Sender: camm@ualberta.ca In-Reply-To: <4C83CA5A.7030404@redhat.com> References: <4C83CA5A.7030404@redhat.com> Date: Mon, 6 Sep 2010 10:37:25 -0600 Message-ID: Subject: Re: [Qemu-devel] Guest cannot handle a PCI BAR > 1GB From: Cam Macdonell Content-Type: multipart/alternative; boundary=0016e65ae5a03ab7a0048f99e5d7 List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Avi Kivity Cc: "qemu-devel@nongnu.org Developers" , linux-kernel --0016e65ae5a03ab7a0048f99e5d7 Content-Type: text/plain; charset=ISO-8859-1 On Sun, Sep 5, 2010 at 10:50 AM, Avi Kivity wrote: > On 09/04/2010 01:22 AM, Cam Macdonell wrote: > >> Hi, >> >> I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and >> I get an error in the guest that it is able to find a mem resource for >> a BAR larger than 1GB. I'm using 64-bit BARs. >> >> when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a >> resource (and searches beyond 32-bit values to find it). Here is a >> log from printfs added to the loop that searches the resources from >> find_resource() in kernel/resource.c:363. >> >> > This is a kernel question, not a qemu issue. Copying lkml. > > > trying 'tmp.start' 1000 to >> 'tmp.end' fff >> trying 'tmp.start' 9f400 to >> 'tmp.end' 9f3ff >> trying 'tmp.start' a0000 to >> 'tmp.end' effff >> trying 'tmp.start' 100000 to >> 'tmp.end' fffff >> trying 'tmp.start' dfffd000 to >> 'tmp.end' dfffcfff >> trying 'tmp.start' e0000000 to >> 'tmp.end' efffffff >> trying 'tmp.start' f2000000 to >> 'tmp.end' f1ffffff >> trying 'tmp.start' f2001000 to >> 'tmp.end' f200ffff >> trying 'tmp.start' f2020000 to >> 'tmp.end' f201ffff >> trying 'tmp.start' f2021000 to >> 'tmp.end' f202ffff >> trying 'tmp.start' f2040000 to >> 'tmp.end' f203ffff >> trying 'tmp.start' f2040100 to >> 'tmp.end' febfffff >> trying 'tmp.start' fec00400 to >> 'tmp.end' fffbffff >> trying 'tmp.start' 100000000 to >> 'tmp.end' ffffffff >> trying 'tmp.start' 1a0000000 to >> 'tmp.end' ffffffffffffffff >> pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit] >> pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit] >> (PCI address [0x1c0000000-0x1ffffffff] >> >> and you can see the BAR is successfully assigned. >> >> However, with a 2GB BAR (below), the search fails, but it also never >> searches beyong 32-bits. Again, all that's changed is the size of the >> ivshmem region. >> >> trying 'tmp.start' 1000 to >> 'tmp.end' fff >> trying 'tmp.start' 9f400 to >> 'tmp.end' 9f3ff >> trying 'tmp.start' a0000 to >> 'tmp.end' effff >> trying 'tmp.start' 100000 to >> 'tmp.end' fffff >> trying 'tmp.start' dfffd000 to >> 'tmp.end' dfffcfff >> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) >> >> Is there a limit to PCI BAR sizes or resources? Any pointers or >> further debugging tips are greatly appreciated. >> >> > What kernel version are you looking at? > latest kvm git, 2.6.36-rc2+ > > Please add printks to the loop so we can see this->start and this->end. It > smells like a truncation issue. Success with a 1GB BAR this->start 1000, this->end 9f3ff this->start 9f400, this->end 9ffff this->start f0000, this->end fffff this->start 100000, this->end dfffcfff this->start dfffd000, this->end dfffffff this->start f0000000, this->end f1ffffff this->start f2000000, this->end f2000fff this->start f2010000, this->end f201ffff this->start f2020000, this->end f2020fff this->start f2030000, this->end f203ffff this->start f2040000, this->end f20400ff this->start fec00000, this->end fec003ff this->start fffc0000, this->end ffffffff this->start 100000000, this->end 11fffffff tmp.start 120000000, tmp.end ffffffffffffffff pci 0000:00:04.0: BAR 2: assigned [mem 0x140000000-0x17fffffff 64bit] and when it fails with a 2GB BAR, the following is printed this->start 1000, this->end 9f3ff this->start 9f400, this->end 9ffff this->start f0000, this->end fffff this->start 100000, this->end dfffcfff this->start dfffd000, this->end dfffffff pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) I added a few more debug statements and found that in the failure case, the function returns that it found a region (the last one printed before the error). I've added printfs for the two tests in the if that determine when a region is found: if (tmp.start < tmp.end && tmp.end - tmp.start >= size - 1) { new->start = tmp.start; new->end = tmp.start + size - 1; printk(KERN_INFO "returning 0\n"); return 0; } this->start 1000, this->end 9f3ff tmp.start 80000000, tmp.end fff true: ffffffff80000fff >= 7fffffff this->start 9f400, this->end 9ffff tmp.start 80000000, tmp.end 9f3ff true: ffffffff8009f3ff >= 7fffffff this->start f0000, this->end fffff tmp.start 80000000, tmp.end effff true: ffffffff800effff >= 7fffffff this->start 100000, this->end dfffcfff tmp.start 80000000, tmp.end fffff true: ffffffff800fffff >= 7fffffff this->start dfffd000, this->end dfffffff tmp.start 100000, tmp.end dfffcfff true: 100000 < dfffcfff true: dfefcfff >= 7fffffff returning 0 pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) > > > -- > error compiling committee.c: too many arguments to function > > --0016e65ae5a03ab7a0048f99e5d7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Sun, Sep 5, 2010 at 10:50 AM, Avi Kiv= ity <avi@redhat.com> wrote:
=A0On 09/04/2010 01:22 AM, Cam Macdonell wrote:
Hi,

I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and I get an error in the guest that it is able to find a mem resource for
a BAR larger than 1GB. =A0I'm using 64-bit BARs.

when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a
resource (and searches beyond 32-bit values to find it). =A0Here is a
log from printfs added to the loop that searches the resources from
find_resource() in kernel/resource.c:363.


This is a kernel question, not a qemu issue. =A0Copying lkml.


trying 'tmp.start' 1000 to
=A0 =A0 =A0 =A0 'tmp.end' fff
trying 'tmp.start' 9f400 to
=A0 =A0 =A0 =A0 'tmp.end' 9f3ff
trying 'tmp.start' a0000 to
=A0 =A0 =A0 =A0 'tmp.end' effff
trying 'tmp.start' 100000 to
=A0 =A0 =A0 =A0 'tmp.end' fffff
trying 'tmp.start' dfffd000 to
=A0 =A0 =A0 =A0 'tmp.end' dfffcfff
trying 'tmp.start' e0000000 to
=A0 =A0 =A0 =A0 'tmp.end' efffffff
trying 'tmp.start' f2000000 to
=A0 =A0 =A0 =A0 'tmp.end' f1ffffff
trying 'tmp.start' f2001000 to
=A0 =A0 =A0 =A0 'tmp.end' f200ffff
trying 'tmp.start' f2020000 to
=A0 =A0 =A0 =A0 'tmp.end' f201ffff
trying 'tmp.start' f2021000 to
=A0 =A0 =A0 =A0 'tmp.end' f202ffff
trying 'tmp.start' f2040000 to
=A0 =A0 =A0 =A0 'tmp.end' f203ffff
trying 'tmp.start' f2040100 to
=A0 =A0 =A0 =A0 'tmp.end' febfffff
trying 'tmp.start' fec00400 to
=A0 =A0 =A0 =A0 'tmp.end' fffbffff
trying 'tmp.start' 100000000 to
=A0 =A0 =A0 =A0 'tmp.end' ffffffff
trying 'tmp.start' 1a0000000 to
=A0 =A0 =A0 =A0 'tmp.end' ffffffffffffffff
pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit]
pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit]
(PCI address [0x1c0000000-0x1ffffffff]

and you can see the BAR is successfully assigned.

However, with a 2GB BAR (below), the search fails, but it also never
searches beyong 32-bits. =A0Again, all that's changed is the size of th= e
ivshmem region.

trying 'tmp.start' 1000 to
=A0 =A0 =A0 =A0 'tmp.end' fff
trying 'tmp.start' 9f400 to
=A0 =A0 =A0 =A0 'tmp.end' 9f3ff
trying 'tmp.start' a0000 to
=A0 =A0 =A0 =A0 'tmp.end' effff
trying 'tmp.start' 100000 to
=A0 =A0 =A0 =A0 'tmp.end' fffff
trying 'tmp.start' dfffd000 to
=A0 =A0 =A0 =A0 'tmp.end' dfffcfff
pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)

Is there a limit to PCI BAR sizes or resources? =A0Any pointers or
further debugging tips are greatly appreciated.


What kernel version are you looking at?

latest kvm git, 2.6.36-rc2+
=A0

Please add printks to the loop so we can see this->start and this->en= d. =A0It smells like a truncation issue.

Su= ccess with a 1GB BAR

this->start 1000, thi= s->end 9f3ff
this->start 9f400, this->end 9ffff
this->start f000= 0, this->end fffff
this->start 100000, this->end dfffcff= f
this->start dfffd000, this->end dfffffff
this-&= gt;start f0000000, this->end f1ffffff
this->start f2000000, this->end f2000fff
this->star= t f2010000, this->end f201ffff
this->start f2020000, this-&= gt;end f2020fff
this->start f2030000, this->end f203ffff
this->start f2040000, this->end f20400ff
this->star= t fec00000, this->end fec003ff
this->start fffc0000, this-&= gt;end ffffffff
this->start 100000000, this->end 11fffffff<= /div>
tmp.start 120000000, tmp.end ffffffffffffffff
pci 0000:00:04= .0: BAR 2: assigned [mem 0x140000000-0x17fffffff 64bit]
and when it fails with a 2GB BAR, the following is printed

this->start 1000, this->end 9f3ff
= this->start 9f400, this->end 9ffff
this->start f0000, th= is->end fffff
this->start 100000, this->end dfffcfff
this->start dfffd000, this->end dfffffff
pci 0000:00:0= 4.0: BAR 2: can't assign mem (size 0x80000000)

I added a few more debug statements and found that in the failure ca= se, the function returns that it found a region (the last one printed befor= e the error). =A0I've added printfs for the two tests in the if that de= termine when a region is found:

=A0=A0 =A0 =A0 if (tmp.start < tmp.end &&am= p; tmp.end - tmp.start >=3D size - 1) {
=A0=A0 =A0 =A0 =A0 =A0= =A0new->start =3D tmp.start;
=A0=A0 =A0 =A0 =A0 =A0 =A0new-&g= t;end =3D tmp.start + size - 1;
=A0=A0 =A0 =A0 =A0 =A0 =A0printk(KERN_INFO "returning 0\n");
=A0=A0 =A0 =A0 =A0 =A0 =A0return 0;
=A0=A0 =A0 =A0 =A0} =A0= =A0=A0

this->start 1000, this->e= nd 9f3ff
tmp.start 80000000, tmp.end fff
=A0=A0 =A0true: ffffffff80000fff >=3D 7fffffff
this->s= tart 9f400, this->end 9ffff
tmp.start 80000000, tmp.end 9f3ff<= /div>
=A0=A0 =A0true: ffffffff8009f3ff >=3D 7fffffff
this-= >start f0000, this->end fffff
tmp.start 80000000, tmp.end effff
=A0=A0 =A0true: ffffffff80= 0effff >=3D 7fffffff
this->start 100000, this->end dfffc= fff
tmp.start 80000000, tmp.end fffff
=A0=A0 =A0true: f= fffffff800fffff >=3D 7fffffff
this->start dfffd000, this->end dfffffff
tmp.start 100= 000, tmp.end dfffcfff
=A0=A0 =A0true: 100000 < dfffcfff
<= div>=A0=A0 =A0true: dfefcfff >=3D 7fffffff
returning 0
pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
=



=A0


--
error compiling committee.c: too many arguments to function


--0016e65ae5a03ab7a0048f99e5d7--