[Qemu-devel] viewing continuous guest virtual memory as continuous in qemu

[Qemu-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

Hi,

 I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map.

 AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering.

 Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device?

 I'd appreciate any help,

Alon

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote:
> Hi,
> 

Converting qemu's ram allocation to a mmap and using remap_file_pages seems
like it could work. Any ideas why it wouldn't?

Alon

>  I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map.
> 
>  AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering.
> 
>  Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device?
> 
>  I'd appreciate any help,
> 
> Alon
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Avi Kivity]

On 10/02/2011 04:31 PM, Alon Levy wrote:
> On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote:
> >  Hi,
> >
>
> Converting qemu's ram allocation to a mmap and using remap_file_pages seems
> like it could work. Any ideas why it wouldn't?
>

It's linux-specific.  Also, does it work on anonymous memory?

I suggest using scatter-gather, though it's annoying.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

On Sun, Oct 02, 2011 at 07:12:07PM +0200, Avi Kivity wrote:
> On 10/02/2011 04:31 PM, Alon Levy wrote:
> >On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote:
> >>  Hi,
> >>
> >
> >Converting qemu's ram allocation to a mmap and using remap_file_pages seems
> >like it could work. Any ideas why it wouldn't?
> >
> 
> It's linux-specific.  Also, does it work on anonymous memory?

Anonymous: I don't think so. I've done shm_open and mmap'ed it twice, one for the ram
and the other to do remap_file_pages on it. Couldn't figure out how to get two mmaps pointing
to the same physical memory without shm_open.

> 
> I suggest using scatter-gather, though it's annoying.
> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Yonit Halperin]

On 10/02/2011 03:24 PM, Alon Levy wrote:
> Hi,
>
>   I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map.
>
>   AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering.
>
>   Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device?
>
>   I'd appreciate any help,
>
> Alon
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Hi,
won't there be an overhead for rendering on a non continuous surface? 
Will it be worthwhile comparing to not creating the surface?

BTW. We should test if the split to vram (surfaces) and devram (commands 
and others) is more efficient than having one section. Even if it is 
more efficient, we can remove the split and give to the surfaces higher 
allocation priority on a part of the pci bar.
Anyway, by default, we can try allocating surfaces on the guest RAM. If 
it fails, we can try to allocate on the pci-bar.

Cheers,
Yonit.

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

On Mon, Oct 03, 2011 at 10:17:59AM +0200, Yonit Halperin wrote:
> On 10/02/2011 03:24 PM, Alon Levy wrote:
> >Hi,
> >
> >  I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map.
> >
> >  AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering.
> >
> >  Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device?
> >
> >  I'd appreciate any help,
> >
> >Alon
> >_______________________________________________
> >Spice-devel mailing list
> >Spice-devel@lists.freedesktop.org
> >http://lists.freedesktop.org/mailman/listinfo/spice-devel
> 
> Hi,
> won't there be an overhead for rendering on a non continuous
> surface? Will it be worthwhile comparing to not creating the
> surface?

If I use a scatter-gather list there is overhead of allocating and
copying the surface whenever I want to synchronize. Minimally once
to copy from guest to host, and another copy from host to guest
for any update_area. (we can only copy the required area.

If I use page remapping like remap_file_pages does, I don't think
there is any overhead for rendering. There is overhead for doing
the remap_file_pages calls, but they are minimal (or so the man page
says). I should benchmark this.

The additional cost is not large - I suppose rendering should be more
costly then a memcpy. But the question is true regardless of this -
some surfaces should be punted probably, if we had an oracle to know they
would be immediately update_area'ed and destroyed.

> 
> BTW. We should test if the split to vram (surfaces) and devram
> (commands and others) is more efficient than having one section.
> Even if it is more efficient, we can remove the split and give to
> the surfaces higher allocation priority on a part of the pci bar.
> Anyway, by default, we can try allocating surfaces on the guest RAM.
> If it fails, we can try to allocate on the pci-bar.
> 

Right. What I was aiming at is removing the BAR all together. This reduces
per vm allocation, and we can still ensure a maximum via the driver. It
also reduces PCI requirements, which are a problem with more then one card.

Actually the more productive thing for reducing PCI memory would be to change
to a single card for multiple monitor support. Another reason for allocating
on guest RAM is to make migration simpler (but I'm not sure it really is).

> Cheers,
> Yonit.
> 
> 

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

On Mon, Oct 03, 2011 at 10:37:55AM +0200, Alon Levy wrote:
> On Mon, Oct 03, 2011 at 10:17:59AM +0200, Yonit Halperin wrote:
> > On 10/02/2011 03:24 PM, Alon Levy wrote:
> > >Hi,
> > >
> > >  I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map.
> > >
> > >  AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering.
> > >
> > >  Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device?
> > >
> > >  I'd appreciate any help,
> > >
> > >Alon
> > >_______________________________________________
> > >Spice-devel mailing list
> > >Spice-devel@lists.freedesktop.org
> > >http://lists.freedesktop.org/mailman/listinfo/spice-devel
> > 
> > Hi,
> > won't there be an overhead for rendering on a non continuous
> > surface? Will it be worthwhile comparing to not creating the
> > surface?
> 
> If I use a scatter-gather list there is overhead of allocating and
> copying the surface whenever I want to synchronize. Minimally once
> to copy from guest to host, and another copy from host to guest
> for any update_area. (we can only copy the required area.
> 
> If I use page remapping like remap_file_pages does, I don't think
> there is any overhead for rendering. There is overhead for doing
> the remap_file_pages calls, but they are minimal (or so the man page
> says). I should benchmark this.
> 
> The additional cost is not large - I suppose rendering should be more
> costly then a memcpy. But the question is true regardless of this -
> some surfaces should be punted probably, if we had an oracle to know they
> would be immediately update_area'ed and destroyed.

Actually we could delay pushing the commands to the server until there is a
command that relies on this surface, or an update_area occours. If update_area
happens first, the surface has not been created yet (we only need to store
the commands), and we can do the whole thing on the guest - we can't punt, since
it is too late, but we can create the gdi surface ourselves and replay all
the commands. If a command relying on this surface happens first (BitBlt to
another surface), then we push all the commands to the server. This would play well
with making the command ring hold bunches like the release ring already does, instead
of individual commands.

> 
> > 
> > BTW. We should test if the split to vram (surfaces) and devram
> > (commands and others) is more efficient than having one section.
> > Even if it is more efficient, we can remove the split and give to
> > the surfaces higher allocation priority on a part of the pci bar.
> > Anyway, by default, we can try allocating surfaces on the guest RAM.
> > If it fails, we can try to allocate on the pci-bar.
> > 
> 
> Right. What I was aiming at is removing the BAR all together. This reduces
> per vm allocation, and we can still ensure a maximum via the driver. It
> also reduces PCI requirements, which are a problem with more then one card.
> 
> Actually the more productive thing for reducing PCI memory would be to change
> to a single card for multiple monitor support. Another reason for allocating
> on guest RAM is to make migration simpler (but I'm not sure it really is).
> 
> > Cheers,
> > Yonit.
> > 
> > 
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Avi Kivity]

On 10/03/2011 10:37 AM, Alon Levy wrote:
> >
> >  Hi,
> >  won't there be an overhead for rendering on a non continuous
> >  surface? Will it be worthwhile comparing to not creating the
> >  surface?
>
> If I use a scatter-gather list there is overhead of allocating and
> copying the surface whenever I want to synchronize. Minimally once
> to copy from guest to host, and another copy from host to guest
> for any update_area. (we can only copy the required area.
>
> If I use page remapping like remap_file_pages does, I don't think
> there is any overhead for rendering. There is overhead for doing
> the remap_file_pages calls, but they are minimal (or so the man page
> says). I should benchmark this.

It's not trivial; and kvm and smp magnify the cost.  It shouldn't be 
done as part of normal rendering (setup is okay).

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Gerd Hoffmann]

   Hi,

> AFAIU this works only when the guest allocates a continuous range of
> physical pages. This is a large requirement from the guest, which I'd
> like to drop.

Is it?  The world is moving to huge pages, with all the stuff needed for 
it like moving around userspace pages to compact memory and make huge 
page allocation easier.  I think these days it is alot easier to 
allocate 2M of continuous physical memory than it used to be a few years 
ago.  At least on linux, dunno about windows.

When allocating stuff at boot time (say qxl kms driver) allocating even 
larger chunks shouldn't be a big issue.  And having a single big guest 
memory chunk, then register that as qxl memory slot is what works best 
with the existing interfaces I guess.

Another option we can think about is a 64bit PCI bar for the surfaces 
which can be moved out of the low 4G.

> So I would like to have the guest use a regular
> allocator, generating for instance two sequential pages in virtual
> memory that are scattered in physical memory. Those two physical
> guest page addresses (gp1 and gp2) correspond to two host virtual
> memory addresses (hv1, hv2). I would now like to provide to
> spice-server a single virtual address p that maps to those two pages
> in sequence.

Playing mapping tricks like this doesn't come for free.  When doing it 
this way we probaby still want to register a big chunk of memory as qxl 
memory slot so we have the mapping cost only once, not for each and 
every surface we create and destroy.

cheers,
   Gerd

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Alon Levy]

On Tue, Oct 11, 2011 at 01:28:12PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> >AFAIU this works only when the guest allocates a continuous range of
> >physical pages. This is a large requirement from the guest, which I'd
> >like to drop.
> 
> Is it?  The world is moving to huge pages, with all the stuff needed
> for it like moving around userspace pages to compact memory and make
> huge page allocation easier.  I think these days it is alot easier
> to allocate 2M of continuous physical memory than it used to be a
> few years ago.  At least on linux, dunno about windows.
> 
> When allocating stuff at boot time (say qxl kms driver) allocating
> even larger chunks shouldn't be a big issue.  And having a single
> big guest memory chunk, then register that as qxl memory slot is
> what works best with the existing interfaces I guess.

Right, this would work. I was trying to avoid claiming a large chunk up
front. I was also trying to avoid having our own allocator, although I
think that's not really a problem (can be replaced with an in kernel
allocator probably).

> 
> Another option we can think about is a 64bit PCI bar for the
> surfaces which can be moved out of the low 4G.
> 

I heard this suggested by Avi, so this would allow us to allocate a
large chunk without requiring any memory hole?

> >So I would like to have the guest use a regular
> >allocator, generating for instance two sequential pages in virtual
> >memory that are scattered in physical memory. Those two physical
> >guest page addresses (gp1 and gp2) correspond to two host virtual
> >memory addresses (hv1, hv2). I would now like to provide to
> >spice-server a single virtual address p that maps to those two pages
> >in sequence.
> 
> Playing mapping tricks like this doesn't come for free.  When doing
> it this way we probaby still want to register a big chunk of memory
> as qxl memory slot so we have the mapping cost only once, not for
> each and every surface we create and destroy.
> 
> cheers,
>   Gerd
> 
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu

[Gerd Hoffmann]

   Hi,

>> Another option we can think about is a 64bit PCI bar for the
>> surfaces which can be moved out of the low 4G.
>
> I heard this suggested by Avi, so this would allow us to allocate a
> large chunk without requiring any memory hole?

You still need some address space for it, but as it isn't limited to 
32bit addresses it can be mapped somewhere above 4G, i.e. we don't have 
to squeeze it into the 512MB pci hole at 0xe0000000.  We can easily 
create a qxl gfx card with 1G of vram (aka surface) memory.

Only question is how to handle that in a backward compatible way.  We 
have regions 4+5 in pci config space still unused.  64bit bars need two 
entries, so adding a 64bit bar would use up both.  The new 64bit vram 
bar would cover all vram memory.  The old 32bit vram bar would be a 
window to the first 32 or 64 MB of vram memory.  We could probably even 
create some register to make the 32bit bar window offset configurable, 
so you can access all vram memory using the 32bit bar, just not all at 
the same time.  Not sure the latter is worth the effort with the world 
moving to 64bit.

cheers,
   Gerd