Re: [Qemu-devel] Guest cannot handle a PCI BAR > 1GB

Re: [Qemu-devel] Guest cannot handle a PCI BAR > 1GB

[Avi Kivity]

  On 09/04/2010 01:22 AM, Cam Macdonell wrote:
> Hi,
>
> I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and
> I get an error in the guest that it is able to find a mem resource for
> a BAR larger than 1GB.  I'm using 64-bit BARs.
>
> when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a
> resource (and searches beyond 32-bit values to find it).  Here is a
> log from printfs added to the loop that searches the resources from
> find_resource() in kernel/resource.c:363.
>

This is a kernel question, not a qemu issue.  Copying lkml.

> trying 'tmp.start' 1000 to
>          'tmp.end' fff
> trying 'tmp.start' 9f400 to
>          'tmp.end' 9f3ff
> trying 'tmp.start' a0000 to
>          'tmp.end' effff
> trying 'tmp.start' 100000 to
>          'tmp.end' fffff
> trying 'tmp.start' dfffd000 to
>          'tmp.end' dfffcfff
> trying 'tmp.start' e0000000 to
>          'tmp.end' efffffff
> trying 'tmp.start' f2000000 to
>          'tmp.end' f1ffffff
> trying 'tmp.start' f2001000 to
>          'tmp.end' f200ffff
> trying 'tmp.start' f2020000 to
>          'tmp.end' f201ffff
> trying 'tmp.start' f2021000 to
>          'tmp.end' f202ffff
> trying 'tmp.start' f2040000 to
>          'tmp.end' f203ffff
> trying 'tmp.start' f2040100 to
>          'tmp.end' febfffff
> trying 'tmp.start' fec00400 to
>          'tmp.end' fffbffff
> trying 'tmp.start' 100000000 to
>          'tmp.end' ffffffff
> trying 'tmp.start' 1a0000000 to
>          'tmp.end' ffffffffffffffff
> pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit]
> pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit]
> (PCI address [0x1c0000000-0x1ffffffff]
>
> and you can see the BAR is successfully assigned.
>
> However, with a 2GB BAR (below), the search fails, but it also never
> searches beyong 32-bits.  Again, all that's changed is the size of the
> ivshmem region.
>
> trying 'tmp.start' 1000 to
>          'tmp.end' fff
> trying 'tmp.start' 9f400 to
>          'tmp.end' 9f3ff
> trying 'tmp.start' a0000 to
>          'tmp.end' effff
> trying 'tmp.start' 100000 to
>          'tmp.end' fffff
> trying 'tmp.start' dfffd000 to
>          'tmp.end' dfffcfff
> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
>
> Is there a limit to PCI BAR sizes or resources?  Any pointers or
> further debugging tips are greatly appreciated.
>

What kernel version are you looking at?

Please add printks to the loop so we can see this->start and this->end.  
It smells like a truncation issue.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Guest cannot handle a PCI BAR > 1GB

[Cam Macdonell]

On Mon, Sep 6, 2010 at 10:37 AM, Cam Macdonell <cam@cs.ualberta.ca> wrote:
>
>
> On Sun, Sep 5, 2010 at 10:50 AM, Avi Kivity <avi@redhat.com> wrote:
>>
>>  On 09/04/2010 01:22 AM, Cam Macdonell wrote:
>>>
>>> Hi,
>>>
>>> I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and
>>> I get an error in the guest that it is able to find a mem resource for
>>> a BAR larger than 1GB.  I'm using 64-bit BARs.
>>>
>>> when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a
>>> resource (and searches beyond 32-bit values to find it).  Here is a
>>> log from printfs added to the loop that searches the resources from
>>> find_resource() in kernel/resource.c:363.
>>>
>>
>> This is a kernel question, not a qemu issue.  Copying lkml.
>>
>>> trying 'tmp.start' 1000 to
>>>         'tmp.end' fff
>>> trying 'tmp.start' 9f400 to
>>>         'tmp.end' 9f3ff
>>> trying 'tmp.start' a0000 to
>>>         'tmp.end' effff
>>> trying 'tmp.start' 100000 to
>>>         'tmp.end' fffff
>>> trying 'tmp.start' dfffd000 to
>>>         'tmp.end' dfffcfff
>>> trying 'tmp.start' e0000000 to
>>>         'tmp.end' efffffff
>>> trying 'tmp.start' f2000000 to
>>>         'tmp.end' f1ffffff
>>> trying 'tmp.start' f2001000 to
>>>         'tmp.end' f200ffff
>>> trying 'tmp.start' f2020000 to
>>>         'tmp.end' f201ffff
>>> trying 'tmp.start' f2021000 to
>>>         'tmp.end' f202ffff
>>> trying 'tmp.start' f2040000 to
>>>         'tmp.end' f203ffff
>>> trying 'tmp.start' f2040100 to
>>>         'tmp.end' febfffff
>>> trying 'tmp.start' fec00400 to
>>>         'tmp.end' fffbffff
>>> trying 'tmp.start' 100000000 to
>>>         'tmp.end' ffffffff
>>> trying 'tmp.start' 1a0000000 to
>>>         'tmp.end' ffffffffffffffff
>>> pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit]
>>> pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit]
>>> (PCI address [0x1c0000000-0x1ffffffff]
>>>
>>> and you can see the BAR is successfully assigned.
>>>
>>> However, with a 2GB BAR (below), the search fails, but it also never
>>> searches beyong 32-bits.  Again, all that's changed is the size of the
>>> ivshmem region.
>>>
>>> trying 'tmp.start' 1000 to
>>>         'tmp.end' fff
>>> trying 'tmp.start' 9f400 to
>>>         'tmp.end' 9f3ff
>>> trying 'tmp.start' a0000 to
>>>         'tmp.end' effff
>>> trying 'tmp.start' 100000 to
>>>         'tmp.end' fffff
>>> trying 'tmp.start' dfffd000 to
>>>         'tmp.end' dfffcfff
>>> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
>>>
>>> Is there a limit to PCI BAR sizes or resources?  Any pointers or
>>> further debugging tips are greatly appreciated.
>>>
>>
>> What kernel version are you looking at?
>
> latest kvm git, 2.6.36-rc2+
>
>>
>> Please add printks to the loop so we can see this->start and this->end.  It smells like a truncation issue.
>
> Success with a 1GB BAR
> this->start 1000, this->end 9f3ff
> this->start 9f400, this->end 9ffff
> this->start f0000, this->end fffff
> this->start 100000, this->end dfffcfff
> this->start dfffd000, this->end dfffffff
> this->start f0000000, this->end f1ffffff
> this->start f2000000, this->end f2000fff
> this->start f2010000, this->end f201ffff
> this->start f2020000, this->end f2020fff
> this->start f2030000, this->end f203ffff
> this->start f2040000, this->end f20400ff
> this->start fec00000, this->end fec003ff
> this->start fffc0000, this->end ffffffff
> this->start 100000000, this->end 11fffffff
> tmp.start 120000000, tmp.end ffffffffffffffff
> pci 0000:00:04.0: BAR 2: assigned [mem 0x140000000-0x17fffffff 64bit]
> and when it fails with a 2GB BAR, the following is printed
> this->start 1000, this->end 9f3ff
> this->start 9f400, this->end 9ffff
> this->start f0000, this->end fffff
> this->start 100000, this->end dfffcfff
> this->start dfffd000, this->end dfffffff
> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
> I added a few more debug statements and found that in the failure case, the function returns that it found a region (the last one printed before the error).  I've added printfs for the two tests in the if that determine when a region is found:
>        if (tmp.start < tmp.end && tmp.end - tmp.start >= size - 1) {
>             new->start = tmp.start;
>             new->end = tmp.start + size - 1;
>             printk(KERN_INFO "returning 0\n");
>             return 0;
>         }
> this->start 1000, this->end 9f3ff
> tmp.start 80000000, tmp.end fff
>     true: ffffffff80000fff >= 7fffffff
> this->start 9f400, this->end 9ffff
> tmp.start 80000000, tmp.end 9f3ff
>     true: ffffffff8009f3ff >= 7fffffff
> this->start f0000, this->end fffff
> tmp.start 80000000, tmp.end effff
>     true: ffffffff800effff >= 7fffffff
> this->start 100000, this->end dfffcfff
> tmp.start 80000000, tmp.end fffff
>     true: ffffffff800fffff >= 7fffffff
> this->start dfffd000, this->end dfffffff
> tmp.start 100000, tmp.end dfffcfff
>     true: 100000 < dfffcfff
>     true: dfefcfff >= 7fffffff
> returning 0
> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)

Further to this, it seems tmp.start is getting set to zero by the ALIGN macro

2GB BAR:
this->start dfffd000, this->end dfffffff
tmp.start dfffd000
tmp.start 0
tmp.start 100000, tmp.end dfffcfff

1GB BAR:
this->start dfffd000, this->end dfffffff
tmp.start dfffd000
tmp.start 100000000
tmp.start 100000000, tmp.end dfffcfff

I'll dig into the ALIGN macro.

>
>
>
>
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>

Re: [Qemu-devel] [solved] Guest cannot handle a PCI BAR > 1GB

[Cam Macdonell]

It seems it was the alignment value being passed back from
pci_resource_alignment().  The return type is an int, which was
causing value of 2GB to be sign extended to to 0xffffffff80000000.
Changing the return type to resource_size_t allows BAR values >= 2GB
to be successfully assigned.

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 679c39d..3d23522 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -318,7 +318,7 @@ static inline int pci_ats_enabled(struct pci_dev *dev)
 }
 #endif /* CONFIG_PCI_IOV */

-static inline int pci_resource_alignment(struct pci_dev *dev,
+static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
                                         struct resource *res)
 {
 #ifdef CONFIG_PCI_IOV


On Mon, Sep 6, 2010 at 11:24 AM, Cam Macdonell <cam@cs.ualberta.ca> wrote:
> On Mon, Sep 6, 2010 at 10:37 AM, Cam Macdonell <cam@cs.ualberta.ca> wrote:
>>
>>
>> On Sun, Sep 5, 2010 at 10:50 AM, Avi Kivity <avi@redhat.com> wrote:
>>>
>>>  On 09/04/2010 01:22 AM, Cam Macdonell wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and
>>>> I get an error in the guest that it is able to find a mem resource for
>>>> a BAR larger than 1GB.  I'm using 64-bit BARs.
>>>>
>>>> when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a
>>>> resource (and searches beyond 32-bit values to find it).  Here is a
>>>> log from printfs added to the loop that searches the resources from
>>>> find_resource() in kernel/resource.c:363.
>>>>
>>>
>>> This is a kernel question, not a qemu issue.  Copying lkml.
>>>
>>>> trying 'tmp.start' 1000 to
>>>>         'tmp.end' fff
>>>> trying 'tmp.start' 9f400 to
>>>>         'tmp.end' 9f3ff
>>>> trying 'tmp.start' a0000 to
>>>>         'tmp.end' effff
>>>> trying 'tmp.start' 100000 to
>>>>         'tmp.end' fffff
>>>> trying 'tmp.start' dfffd000 to
>>>>         'tmp.end' dfffcfff
>>>> trying 'tmp.start' e0000000 to
>>>>         'tmp.end' efffffff
>>>> trying 'tmp.start' f2000000 to
>>>>         'tmp.end' f1ffffff
>>>> trying 'tmp.start' f2001000 to
>>>>         'tmp.end' f200ffff
>>>> trying 'tmp.start' f2020000 to
>>>>         'tmp.end' f201ffff
>>>> trying 'tmp.start' f2021000 to
>>>>         'tmp.end' f202ffff
>>>> trying 'tmp.start' f2040000 to
>>>>         'tmp.end' f203ffff
>>>> trying 'tmp.start' f2040100 to
>>>>         'tmp.end' febfffff
>>>> trying 'tmp.start' fec00400 to
>>>>         'tmp.end' fffbffff
>>>> trying 'tmp.start' 100000000 to
>>>>         'tmp.end' ffffffff
>>>> trying 'tmp.start' 1a0000000 to
>>>>         'tmp.end' ffffffffffffffff
>>>> pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit]
>>>> pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit]
>>>> (PCI address [0x1c0000000-0x1ffffffff]
>>>>
>>>> and you can see the BAR is successfully assigned.
>>>>
>>>> However, with a 2GB BAR (below), the search fails, but it also never
>>>> searches beyong 32-bits.  Again, all that's changed is the size of the
>>>> ivshmem region.
>>>>
>>>> trying 'tmp.start' 1000 to
>>>>         'tmp.end' fff
>>>> trying 'tmp.start' 9f400 to
>>>>         'tmp.end' 9f3ff
>>>> trying 'tmp.start' a0000 to
>>>>         'tmp.end' effff
>>>> trying 'tmp.start' 100000 to
>>>>         'tmp.end' fffff
>>>> trying 'tmp.start' dfffd000 to
>>>>         'tmp.end' dfffcfff
>>>> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
>>>>
>>>> Is there a limit to PCI BAR sizes or resources?  Any pointers or
>>>> further debugging tips are greatly appreciated.
>>>>
>>>
>>> What kernel version are you looking at?
>>
>> latest kvm git, 2.6.36-rc2+
>>
>>>
>>> Please add printks to the loop so we can see this->start and this->end.  It smells like a truncation issue.
>>
>> Success with a 1GB BAR
>> this->start 1000, this->end 9f3ff
>> this->start 9f400, this->end 9ffff
>> this->start f0000, this->end fffff
>> this->start 100000, this->end dfffcfff
>> this->start dfffd000, this->end dfffffff
>> this->start f0000000, this->end f1ffffff
>> this->start f2000000, this->end f2000fff
>> this->start f2010000, this->end f201ffff
>> this->start f2020000, this->end f2020fff
>> this->start f2030000, this->end f203ffff
>> this->start f2040000, this->end f20400ff
>> this->start fec00000, this->end fec003ff
>> this->start fffc0000, this->end ffffffff
>> this->start 100000000, this->end 11fffffff
>> tmp.start 120000000, tmp.end ffffffffffffffff
>> pci 0000:00:04.0: BAR 2: assigned [mem 0x140000000-0x17fffffff 64bit]
>> and when it fails with a 2GB BAR, the following is printed
>> this->start 1000, this->end 9f3ff
>> this->start 9f400, this->end 9ffff
>> this->start f0000, this->end fffff
>> this->start 100000, this->end dfffcfff
>> this->start dfffd000, this->end dfffffff
>> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
>> I added a few more debug statements and found that in the failure case, the function returns that it found a region (the last one printed before the error).  I've added printfs for the two tests in the if that determine when a region is found:
>>        if (tmp.start < tmp.end && tmp.end - tmp.start >= size - 1) {
>>             new->start = tmp.start;
>>             new->end = tmp.start + size - 1;
>>             printk(KERN_INFO "returning 0\n");
>>             return 0;
>>         }
>> this->start 1000, this->end 9f3ff
>> tmp.start 80000000, tmp.end fff
>>     true: ffffffff80000fff >= 7fffffff
>> this->start 9f400, this->end 9ffff
>> tmp.start 80000000, tmp.end 9f3ff
>>     true: ffffffff8009f3ff >= 7fffffff
>> this->start f0000, this->end fffff
>> tmp.start 80000000, tmp.end effff
>>     true: ffffffff800effff >= 7fffffff
>> this->start 100000, this->end dfffcfff
>> tmp.start 80000000, tmp.end fffff
>>     true: ffffffff800fffff >= 7fffffff
>> this->start dfffd000, this->end dfffffff
>> tmp.start 100000, tmp.end dfffcfff
>>     true: 100000 < dfffcfff
>>     true: dfefcfff >= 7fffffff
>> returning 0
>> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000)
>
> Further to this, it seems tmp.start is getting set to zero by the ALIGN macro
>
> 2GB BAR:
> this->start dfffd000, this->end dfffffff
> tmp.start dfffd000
> tmp.start 0
> tmp.start 100000, tmp.end dfffcfff
>
> 1GB BAR:
> this->start dfffd000, this->end dfffffff
> tmp.start dfffd000
> tmp.start 100000000
> tmp.start 100000000, tmp.end dfffcfff
>
> I'll dig into the ALIGN macro.
>
>>
>>
>>
>>
>>>
>>> --
>>> error compiling committee.c: too many arguments to function
>>>
>>
>

[PATCH] pci: fix pci_resource_alignment prototype

[Chris Wright]

From: Cam Macdonell <cam@cs.ualberta.ca>

* Cam Macdonell (cam@cs.ualberta.ca) wrote:
> It seems it was the alignment value being passed back from
> pci_resource_alignment().  The return type is an int, which was
> causing value of 2GB to be sign extended to to 0xffffffff80000000.
> Changing the return type to resource_size_t allows BAR values >= 2GB
> to be successfully assigned.
<snip>
> -static inline int pci_resource_alignment(struct pci_dev *dev,
> +static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
>                                          struct resource *res)

Yes, that's my mistake.  Thanks for debugging the issue Cam.
This fixes the prototype for both pci_resource_alignment() and
pci_sriov_resource_alignment().

Patch started as debugging effort from Cam Macdonell.

Cc: Cam Macdonell <cam@cs.ualberta.ca>
Cc: Avi Kivity <avi@redhat.com>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
[chrisw: add iov bits]
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
---
 drivers/pci/iov.c |    2 +-
 drivers/pci/pci.h |    5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index ce6a366..553d8ee 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -608,7 +608,7 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno,
  * the VF BAR size multiplied by the number of VFs.  The alignment
  * is just the VF BAR size.
  */
-int pci_sriov_resource_alignment(struct pci_dev *dev, int resno)
+resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno)
 {
 	struct resource tmp;
 	enum pci_bar_type type;
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 679c39d..5d0aeb1 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -262,7 +262,8 @@ extern int pci_iov_init(struct pci_dev *dev);
 extern void pci_iov_release(struct pci_dev *dev);
 extern int pci_iov_resource_bar(struct pci_dev *dev, int resno,
 				enum pci_bar_type *type);
-extern int pci_sriov_resource_alignment(struct pci_dev *dev, int resno);
+extern resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev,
+						    int resno);
 extern void pci_restore_iov_state(struct pci_dev *dev);
 extern int pci_iov_bus_range(struct pci_bus *bus);
 
@@ -318,7 +319,7 @@ static inline int pci_ats_enabled(struct pci_dev *dev)
 }
 #endif /* CONFIG_PCI_IOV */
 
-static inline int pci_resource_alignment(struct pci_dev *dev,
+static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
 					 struct resource *res)
 {
 #ifdef CONFIG_PCI_IOV

Re: [PATCH] pci: fix pci_resource_alignment prototype

[Jesse Barnes]

On Tue, 7 Sep 2010 17:25:20 -0700
Chris Wright <chrisw@sous-sol.org> wrote:

> From: Cam Macdonell <cam@cs.ualberta.ca>
> 
> * Cam Macdonell (cam@cs.ualberta.ca) wrote:
> > It seems it was the alignment value being passed back from
> > pci_resource_alignment().  The return type is an int, which was
> > causing value of 2GB to be sign extended to to 0xffffffff80000000.
> > Changing the return type to resource_size_t allows BAR values >= 2GB
> > to be successfully assigned.
> <snip>
> > -static inline int pci_resource_alignment(struct pci_dev *dev,
> > +static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
> >                                          struct resource *res)
> 
> Yes, that's my mistake.  Thanks for debugging the issue Cam.
> This fixes the prototype for both pci_resource_alignment() and
> pci_sriov_resource_alignment().
> 
> Patch started as debugging effort from Cam Macdonell.
> 
> Cc: Cam Macdonell <cam@cs.ualberta.ca>
> Cc: Avi Kivity <avi@redhat.com>
> Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
> [chrisw: add iov bits]
> Signed-off-by: Chris Wright <chrisw@sous-sol.org>
> ---
>  drivers/pci/iov.c |    2 +-
>  drivers/pci/pci.h |    5 +++--
>  2 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index ce6a366..553d8ee 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -608,7 +608,7 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno,
>   * the VF BAR size multiplied by the number of VFs.  The alignment
>   * is just the VF BAR size.
>   */
> -int pci_sriov_resource_alignment(struct pci_dev *dev, int resno)
> +resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno)
>  {
>  	struct resource tmp;
>  	enum pci_bar_type type;
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 679c39d..5d0aeb1 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -262,7 +262,8 @@ extern int pci_iov_init(struct pci_dev *dev);
>  extern void pci_iov_release(struct pci_dev *dev);
>  extern int pci_iov_resource_bar(struct pci_dev *dev, int resno,
>  				enum pci_bar_type *type);
> -extern int pci_sriov_resource_alignment(struct pci_dev *dev, int resno);
> +extern resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev,
> +						    int resno);
>  extern void pci_restore_iov_state(struct pci_dev *dev);
>  extern int pci_iov_bus_range(struct pci_bus *bus);
>  
> @@ -318,7 +319,7 @@ static inline int pci_ats_enabled(struct pci_dev *dev)
>  }
>  #endif /* CONFIG_PCI_IOV */
>  
> -static inline int pci_resource_alignment(struct pci_dev *dev,
> +static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
>  					 struct resource *res)
>  {
>  #ifdef CONFIG_PCI_IOV
> 
> 

Applied to my for-linus branch, thanks.

-- 
Jesse Barnes, Intel Open Source Technology Center