* [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs
@ 2017-03-11 10:37 Mark Cave-Ayland
2017-03-12 1:46 ` Michael S. Tsirkin
0 siblings, 1 reply; 5+ messages in thread
From: Mark Cave-Ayland @ 2017-03-11 10:37 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S. Tsirkin, Paolo Bonzini
I've been looking at an issue with virtio BARs being mapped incorrectly
on qemu-system-sparc64, and as part of this investigation found that
some 64-bit PCI BAR memory regions don't appear correctly in "info mtree".
The easiest way to see this is to launch QEMU as below and then check
the output of "info mtree":
./qemu-system-sparc64 -device virtio-net-pci
The interesting part is this:
> address-space: memory
> 0000000000000000-ffffffffffffffff (prio 0, i/o): system
> 0000000000000000-0000000007ffffff (prio 0, ram): sun4u.ram
> 000001fe00000000-000001fe0000ffff (prio 0, i/o): apb-config
> 000001fe01000000-000001fe01ffffff (prio 0, i/o): apb-pci-config
> 000001fe02000000-000001fe0200ffff (prio 0, i/o): alias apb-pci ioport @io 0000000000000000-000000000000ffff
> 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> 000001ff000a0000-000001ff000affff (prio 2, i/o): alias vga.chain4 @vga.vram 0000000000000000-000000000000ffff
> 000001ff000a0000-000001ff000bffff (prio 1, i/o): vga-lowmem
> 000001ff01000000-000001ff01ffffff (prio 1, ram): vga.vram
> 000001ff02000000-000001ff02000fff (prio 1, i/o): vga.mmio
> 000001ff02000400-000001ff0200041f (prio 0, i/o): vga ioports remapped
> 000001ff02000500-000001ff02000515 (prio 0, i/o): bochs dispi interface
> 000001ff02000600-000001ff02000607 (prio 0, i/o): qemu extended regs
> 000001ff02010000-000001ff0201ffff (prio 1, rom): vga.rom
> 000001ff03000000-000001ff03ffffff (prio 1, i/o): alias bar0 @io 0000000000000000-0000000000ffffff
> 000001ff04000000-000001ff0403ffff (prio 1, rom): ne2000.rom
> 000001ff04080000-000001ff040bffff (prio 1, rom): virtio-net-pci.rom
> 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here virtio-pci is mapped to address 0x1fe04040000 even though its
parent pci-mmio memory region is at 0x1ff00000000.
> 000001fe04040000-000001fe04040fff (prio 0, i/o): virtio-pci-common
> 000001fe04041000-000001fe04041fff (prio 0, i/o): virtio-pci-isr
> 000001fe04042000-000001fe04042fff (prio 0, i/o): virtio-pci-device
> 000001fe04043000-000001fe04043fff (prio 0, i/o): virtio-pci-notify
> 000001fff0000000-000001fff03fffff (prio 0, rom): sun4u.prom
The bug in the current OpenBIOS is that 64-bit BARs are incorrectly
identified, and so the BAR MSB register is set to 0xffffffff to detect
the size as per a standard 32-bit BAR. At this point OpenBIOS detects
something is wrong and aborts, leaving the 64-bit BAR set to 0xffffffff
0x04040000.
This is accurately reflected in the virtio-pci memory region itself, i.e.
> memory-region: virtio-pci
> ffffffff04040000-ffffffff04043fff (prio 1, i/o): virtio-pci
> ffffffff04040000-ffffffff04040fff (prio 0, i/o): virtio-pci-common
> ffffffff04041000-ffffffff04041fff (prio 0, i/o): virtio-pci-isr
> ffffffff04042000-ffffffff04042fff (prio 0, i/o): virtio-pci-device
> ffffffff04043000-ffffffff04043fff (prio 0, i/o): virtio-pci-notify
But as you can see above the overall effect is that the virtio-pci
device ends up somehow being mapped outside (and indeed, before the
start of) its parent memory region.
Fixing the bug in OpenBIOS itself is fairly trivial, although it took
much longer to diagnose due to the misleading output above. It does seem
that something isn't quite right with the handling of the large 64-bit
BARs in this case, so Michael suggested I email the list for further
comment.
ATB,
Mark.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs
2017-03-11 10:37 [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs Mark Cave-Ayland
@ 2017-03-12 1:46 ` Michael S. Tsirkin
2017-03-12 3:56 ` Michael S. Tsirkin
0 siblings, 1 reply; 5+ messages in thread
From: Michael S. Tsirkin @ 2017-03-12 1:46 UTC (permalink / raw)
To: Mark Cave-Ayland; +Cc: qemu-devel, Paolo Bonzini
On Sat, Mar 11, 2017 at 10:37:09AM +0000, Mark Cave-Ayland wrote:
> I've been looking at an issue with virtio BARs being mapped incorrectly
> on qemu-system-sparc64, and as part of this investigation found that
> some 64-bit PCI BAR memory regions don't appear correctly in "info mtree".
>
> The easiest way to see this is to launch QEMU as below and then check
> the output of "info mtree":
>
> ./qemu-system-sparc64 -device virtio-net-pci
>
> The interesting part is this:
>
> > address-space: memory
> > 0000000000000000-ffffffffffffffff (prio 0, i/o): system
> > 0000000000000000-0000000007ffffff (prio 0, ram): sun4u.ram
> > 000001fe00000000-000001fe0000ffff (prio 0, i/o): apb-config
> > 000001fe01000000-000001fe01ffffff (prio 0, i/o): apb-pci-config
> > 000001fe02000000-000001fe0200ffff (prio 0, i/o): alias apb-pci ioport @io 0000000000000000-000000000000ffff
> > 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff000a0000-000001ff000affff (prio 2, i/o): alias vga.chain4 @vga.vram 0000000000000000-000000000000ffff
> > 000001ff000a0000-000001ff000bffff (prio 1, i/o): vga-lowmem
> > 000001ff01000000-000001ff01ffffff (prio 1, ram): vga.vram
> > 000001ff02000000-000001ff02000fff (prio 1, i/o): vga.mmio
> > 000001ff02000400-000001ff0200041f (prio 0, i/o): vga ioports remapped
> > 000001ff02000500-000001ff02000515 (prio 0, i/o): bochs dispi interface
> > 000001ff02000600-000001ff02000607 (prio 0, i/o): qemu extended regs
> > 000001ff02010000-000001ff0201ffff (prio 1, rom): vga.rom
> > 000001ff03000000-000001ff03ffffff (prio 1, i/o): alias bar0 @io 0000000000000000-0000000000ffffff
> > 000001ff04000000-000001ff0403ffff (prio 1, rom): ne2000.rom
> > 000001ff04080000-000001ff040bffff (prio 1, rom): virtio-net-pci.rom
> > 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Here virtio-pci is mapped to address 0x1fe04040000 even though its
> parent pci-mmio memory region is at 0x1ff00000000.
>
> > 000001fe04040000-000001fe04040fff (prio 0, i/o): virtio-pci-common
> > 000001fe04041000-000001fe04041fff (prio 0, i/o): virtio-pci-isr
> > 000001fe04042000-000001fe04042fff (prio 0, i/o): virtio-pci-device
> > 000001fe04043000-000001fe04043fff (prio 0, i/o): virtio-pci-notify
> > 000001fff0000000-000001fff03fffff (prio 0, rom): sun4u.prom
>
> The bug in the current OpenBIOS is that 64-bit BARs are incorrectly
> identified, and so the BAR MSB register is set to 0xffffffff to detect
> the size as per a standard 32-bit BAR. At this point OpenBIOS detects
> something is wrong and aborts, leaving the 64-bit BAR set to 0xffffffff
> 0x04040000.
>
> This is accurately reflected in the virtio-pci memory region itself, i.e.
>
> > memory-region: virtio-pci
> > ffffffff04040000-ffffffff04043fff (prio 1, i/o): virtio-pci
> > ffffffff04040000-ffffffff04040fff (prio 0, i/o): virtio-pci-common
> > ffffffff04041000-ffffffff04041fff (prio 0, i/o): virtio-pci-isr
> > ffffffff04042000-ffffffff04042fff (prio 0, i/o): virtio-pci-device
> > ffffffff04043000-ffffffff04043fff (prio 0, i/o): virtio-pci-notify
>
> But as you can see above the overall effect is that the virtio-pci
> device ends up somehow being mapped outside (and indeed, before the
> start of) its parent memory region.
And specifically we have:
000001ff00000000 + ffffffff04040000 = 000001fe04043000
which seems to be where this is coming from.
It does have to indicate some kind of modelling problem:
since it looks like on sparc BAR is an offset within
PCI MMIO region, I would expect an offset value bigger than
the region size not to be visible.
Specifically documentation states:
- all direct subregions of the root region are matched against the address, in
descending priority order
- if the address lies outside the region offset/size, the subregion is
discarded
> Fixing the bug in OpenBIOS itself is fairly trivial, although it took
> much longer to diagnose due to the misleading output above. It does seem
> that something isn't quite right with the handling of the large 64-bit
> BARs in this case, so Michael suggested I email the list for further
> comment.
>
>
> ATB,
>
> Mark.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs
2017-03-12 1:46 ` Michael S. Tsirkin
@ 2017-03-12 3:56 ` Michael S. Tsirkin
2017-03-12 11:16 ` Mark Cave-Ayland
0 siblings, 1 reply; 5+ messages in thread
From: Michael S. Tsirkin @ 2017-03-12 3:56 UTC (permalink / raw)
To: Mark Cave-Ayland; +Cc: qemu-devel, Paolo Bonzini
On Sun, Mar 12, 2017 at 03:46:57AM +0200, Michael S. Tsirkin wrote:
> On Sat, Mar 11, 2017 at 10:37:09AM +0000, Mark Cave-Ayland wrote:
> > I've been looking at an issue with virtio BARs being mapped incorrectly
> > on qemu-system-sparc64, and as part of this investigation found that
> > some 64-bit PCI BAR memory regions don't appear correctly in "info mtree".
> >
> > The easiest way to see this is to launch QEMU as below and then check
> > the output of "info mtree":
> >
> > ./qemu-system-sparc64 -device virtio-net-pci
> >
> > The interesting part is this:
> >
> > > address-space: memory
> > > 0000000000000000-ffffffffffffffff (prio 0, i/o): system
> > > 0000000000000000-0000000007ffffff (prio 0, ram): sun4u.ram
> > > 000001fe00000000-000001fe0000ffff (prio 0, i/o): apb-config
> > > 000001fe01000000-000001fe01ffffff (prio 0, i/o): apb-pci-config
> > > 000001fe02000000-000001fe0200ffff (prio 0, i/o): alias apb-pci ioport @io 0000000000000000-000000000000ffff
> > > 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> > > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > > 000001ff000a0000-000001ff000affff (prio 2, i/o): alias vga.chain4 @vga.vram 0000000000000000-000000000000ffff
> > > 000001ff000a0000-000001ff000bffff (prio 1, i/o): vga-lowmem
> > > 000001ff01000000-000001ff01ffffff (prio 1, ram): vga.vram
> > > 000001ff02000000-000001ff02000fff (prio 1, i/o): vga.mmio
> > > 000001ff02000400-000001ff0200041f (prio 0, i/o): vga ioports remapped
> > > 000001ff02000500-000001ff02000515 (prio 0, i/o): bochs dispi interface
> > > 000001ff02000600-000001ff02000607 (prio 0, i/o): qemu extended regs
> > > 000001ff02010000-000001ff0201ffff (prio 1, rom): vga.rom
> > > 000001ff03000000-000001ff03ffffff (prio 1, i/o): alias bar0 @io 0000000000000000-0000000000ffffff
> > > 000001ff04000000-000001ff0403ffff (prio 1, rom): ne2000.rom
> > > 000001ff04080000-000001ff040bffff (prio 1, rom): virtio-net-pci.rom
> > > 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > Here virtio-pci is mapped to address 0x1fe04040000 even though its
> > parent pci-mmio memory region is at 0x1ff00000000.
> >
> > > 000001fe04040000-000001fe04040fff (prio 0, i/o): virtio-pci-common
> > > 000001fe04041000-000001fe04041fff (prio 0, i/o): virtio-pci-isr
> > > 000001fe04042000-000001fe04042fff (prio 0, i/o): virtio-pci-device
> > > 000001fe04043000-000001fe04043fff (prio 0, i/o): virtio-pci-notify
> > > 000001fff0000000-000001fff03fffff (prio 0, rom): sun4u.prom
> >
> > The bug in the current OpenBIOS is that 64-bit BARs are incorrectly
> > identified, and so the BAR MSB register is set to 0xffffffff to detect
> > the size as per a standard 32-bit BAR. At this point OpenBIOS detects
> > something is wrong and aborts, leaving the 64-bit BAR set to 0xffffffff
> > 0x04040000.
> >
> > This is accurately reflected in the virtio-pci memory region itself, i.e.
> >
> > > memory-region: virtio-pci
> > > ffffffff04040000-ffffffff04043fff (prio 1, i/o): virtio-pci
> > > ffffffff04040000-ffffffff04040fff (prio 0, i/o): virtio-pci-common
> > > ffffffff04041000-ffffffff04041fff (prio 0, i/o): virtio-pci-isr
> > > ffffffff04042000-ffffffff04042fff (prio 0, i/o): virtio-pci-device
> > > ffffffff04043000-ffffffff04043fff (prio 0, i/o): virtio-pci-notify
> >
> > But as you can see above the overall effect is that the virtio-pci
> > device ends up somehow being mapped outside (and indeed, before the
> > start of) its parent memory region.
>
> And specifically we have:
>
> 000001ff00000000 + ffffffff04040000 = 000001fe04043000
>
> which seems to be where this is coming from.
>
> It does have to indicate some kind of modelling problem:
> since it looks like on sparc BAR is an offset within
> PCI MMIO region, I would expect an offset value bigger than
> the region size not to be visible.
>
> Specifically documentation states:
>
> - all direct subregions of the root region are matched against the address, in
> descending priority order
> - if the address lies outside the region offset/size, the subregion is
> discarded
>
After looking at it some more, I think the issue is merely with how info
mtree presents information, which confuses instead of helping when
overlap triggers. Specifically
000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
...
000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
really means that virtio-pci is not visible at all, this
happens because it starts at offset ffffffff04040000 which is
outside the parent.
I think that the cleanest fix is probably to show 128 bit addresses,
then user will see the real addresses:
000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
...
1000001fe04040000-1000001fe04043fff (prio 1, i/o): virtio-pci
and now it's clear what is going on: virtio-pci is outside pci-mmio.
This would have pointed Mark in the right direction earlier.
Thoughts? Patch?
--
MST
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs
2017-03-12 3:56 ` Michael S. Tsirkin
@ 2017-03-12 11:16 ` Mark Cave-Ayland
2017-03-13 12:02 ` Michael S. Tsirkin
0 siblings, 1 reply; 5+ messages in thread
From: Mark Cave-Ayland @ 2017-03-12 11:16 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Paolo Bonzini, qemu-devel
On 12/03/17 03:56, Michael S. Tsirkin wrote:
> After looking at it some more, I think the issue is merely with how info
> mtree presents information, which confuses instead of helping when
> overlap triggers. Specifically
> 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> ...
> 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
>
> really means that virtio-pci is not visible at all, this
> happens because it starts at offset ffffffff04040000 which is
> outside the parent.
>
> I think that the cleanest fix is probably to show 128 bit addresses,
> then user will see the real addresses:
>
> 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> ...
> 1000001fe04040000-1000001fe04043fff (prio 1, i/o): virtio-pci
>
> and now it's clear what is going on: virtio-pci is outside pci-mmio.
>
> This would have pointed Mark in the right direction earlier.
>
> Thoughts? Patch?
Presumably if someone tried to do this on real hardware, the BAR address
would lie outside of the pci-mmio region and effectively isn't mapped?
If this is the case I'd be happy with a simple qemu_log() showing the
full 64-bit address and region name explaining that it couldn't be
mapped underneath its parent because it was outside its parent region,
and skip the mapping.
ATB,
Mark.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs
2017-03-12 11:16 ` Mark Cave-Ayland
@ 2017-03-13 12:02 ` Michael S. Tsirkin
0 siblings, 0 replies; 5+ messages in thread
From: Michael S. Tsirkin @ 2017-03-13 12:02 UTC (permalink / raw)
To: Mark Cave-Ayland; +Cc: Paolo Bonzini, qemu-devel
On Sun, Mar 12, 2017 at 11:16:44AM +0000, Mark Cave-Ayland wrote:
> On 12/03/17 03:56, Michael S. Tsirkin wrote:
>
> > After looking at it some more, I think the issue is merely with how info
> > mtree presents information, which confuses instead of helping when
> > overlap triggers. Specifically
> > 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> > ...
> > 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
> >
> > really means that virtio-pci is not visible at all, this
> > happens because it starts at offset ffffffff04040000 which is
> > outside the parent.
> >
> > I think that the cleanest fix is probably to show 128 bit addresses,
> > then user will see the real addresses:
> >
> > 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> > ...
> > 1000001fe04040000-1000001fe04043fff (prio 1, i/o): virtio-pci
> >
> > and now it's clear what is going on: virtio-pci is outside pci-mmio.
> >
> > This would have pointed Mark in the right direction earlier.
> >
> > Thoughts? Patch?
>
> Presumably if someone tried to do this on real hardware, the BAR address
> would lie outside of the pci-mmio region and effectively isn't mapped?
>
> If this is the case I'd be happy with a simple qemu_log() showing the
> full 64-bit address and region name explaining that it couldn't be
> mapped underneath its parent because it was outside its parent region,
> and skip the mapping.
>
>
> ATB,
>
> Mark.
I think that's what happens anyway. The bug is merely the desplay in info mtree
which uses 64 bit math.
--
MST
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-03-13 12:02 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-03-11 10:37 [Qemu-devel] Incorrect memory region address with large 64-bit PCI BARs Mark Cave-Ayland
2017-03-12 1:46 ` Michael S. Tsirkin
2017-03-12 3:56 ` Michael S. Tsirkin
2017-03-12 11:16 ` Mark Cave-Ayland
2017-03-13 12:02 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).