Re: Mapping memory regions on s390

Re: Mapping memory regions on s390

[Cornelia Huck]

On Thu, 21 Feb 2019 17:17:47 +0100
Halil Pasic <pasic@linux.ibm.com> wrote:

> On Thu, 21 Feb 2019 15:46:08 +0000
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> 
> > * Cornelia Huck (cohuck@redhat.com) wrote:  
> > > On Thu, 21 Feb 2019 15:11:50 +0000
> > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > >   
> > > > * Cornelia Huck (cohuck@redhat.com) wrote:  
> > > > > On Thu, 21 Feb 2019 13:39:46 +0000
> > > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > > > >     
> > > > > > * Cornelia Huck (cohuck@redhat.com) wrote:    
> > > > > > > On Mon, 18 Feb 2019 15:41:07 +0000
> > > > > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > > > > > >       
> > > > > > > > * Cornelia Huck (cohuck@redhat.com) wrote:      
> > > > > > >       
> > > > > > > > > > >> We can of course switch the order of mappings
> > > > > > > > > > >>
> > > > > > > > > > >> [0x000000000000000      ]
> > > > > > > > > > >> ... Memory region containing RAM
> > > > > > > > > > >> [ram_size         	]
> > > > > > > > > > >> ... Memory region for memory devices (virtio-pmem, virtio-mem ...)
> > > > > > > > > > >> [maxram_size - ram_size ]
> > > > > > > > > > >> ... Memory region for e.g. special PCI/CCW devices
> > > > > > > > > > >> [                    TBD]
> > > > > > > > > > >>
> > > > > > > > > > >> We can size TBD in a way that we e.g. max out the current page table
> > > > > > > > > > >> size before having to switch to more levels.          
> > > > > > > > > > > 
> > > > > > > > > > > Yes, that's fine to set some upper limit; you've just got to make sure
> > > > > > > > > > > that the hypervisor knows where it can put stuff and if the guest
> > > > > > > > > > > does PCI that it knows where it's allowed to put stuff and as long
> > > > > > > > > > > as the two don't overlap everyone is happy.        
> > > > > > > > > 
> > > > > > > > > Hm... is that an issue for pci? Do we need to care, as s390 uses
> > > > > > > > > special instructions anyway? Or do we want to avoid going through them,
> > > > > > > > > so that the guest can use normal read/write?        
> > > > > > > > 
> > > > > > > > That depends.
> > > > > > > > The stuff we use for virtio-fs we need the shared region to be
> > > > > > > > accessible by the guest via normal instructions because we're using for
> > > > > > > > DAX.  For PCI you might be able to avoid it for most other PCI cases.      
> > > > > > > 
> > > > > > > So,
> > > > > > > - virtio-fs regions need to be accessible like normal memory, so they
> > > > > > >   need to show up in the region labeled 'TBD' above       
> > > > > > 
> > > > > > Yes.
> > > > > >     
> > > > > > >  (it would fine to
> > > > > > >   communicate the 'where' through pci structures)      
> > > > > > 
> > > > > > Hmm, mixing PCI structures into something you're not treating as PCI
> > > > > > seems weird to me.    
> > > > > 
> > > > > I was thinking about the addresses in the TBD area... they need to go
> > > > > through _some_ pci structure, I assume?    
> > > > 
> > > > Well I think it depends how you make it work with CCW;
> > > > 
> > > > if the addresses being assigned are assigned by the host then I believe you should use
> > > > the CCW mechanism you suggested to discover the addresses in the guest.
> > > > 
> > > > If the host is going to allocate a block of PCI space and the guest is
> > > > going to allocate the use within that space and access it with normal
> > > > instructions then I think it should go via PCI.  
> > > 
> > > This is getting a bit confusing... let me try to summarize:
> > > 
> > > - We introduce a special area where shared memory areas are supposed to
> > >   live.
> > > - If a virtio device accessed via ccw defines shared regions, the
> > >   driver can discover them via a new ccw that indicates an address in
> > >   that special area.  
> > 
> > Right.
> >   
> > > - If a virtio device accessed via pci defines shared regions, the
> > >   driver will want to discover them via the same mechanism as on other
> > >   platforms. If I read
> > >   https://lists.oasis-open.org/archives/virtio-comment/201901/msg00003.html
> > >   correctly, this will mean an offset into a BAR. This will be a normal
> > >   pci memory region.  
> > 
> > Right; note that like any other BAR it's upto the guest to write the BAR
> > to select which area of GPA it wants the BAR to map to.
> >   
> > > Now, this sounds to me that we'll have regions in different memory
> > > regions, depending on whether they are accessed via ccw or via pci. Not
> > > sure if that's a problem.  
> > 
> > As long as they stay out of each others way it shouldn't be.
> >   
> 
> I'm not sure. What I'm concerted about is:
> * if it is a virtio-ccw device it would need to access the shared memory via
> normal memory access instructions.

Just for reference, my current draft for the ccw spec looks as follows:

Re: Mapping memory regions on s390

[Dr. David Alan Gilbert]

* David Hildenbrand (david@redhat.com) wrote:
> On 01.03.19 14:45, Cornelia Huck wrote:
> > On Fri, 1 Mar 2019 14:18:21 +0100 (CET)
> > Sebastian Ott <sebott@linux.ibm.com> wrote:
> > 
> >> On Fri, 1 Mar 2019, Cornelia Huck wrote:
> >>> On Fri, 1 Mar 2019 13:14:38 +0100 (CET)
> >>> Sebastian Ott <sebott@linux.ibm.com> wrote:  
> > 
> >>>> The iomap, readb stuff is only usable in a PCI context on s390. But what
> >>>> is the problem here? virtio_ccw knows it's not a PCI driver - it doesn't
> >>>> have to use this stuff..  
> >>>
> >>> The device has to put the shared regions somewhere. If the virtio-pci
> >>> transport is used, it's just a normal area within a BAR (IIUC), and
> >>> should be handled just like normal pci memory.
> >>>
> >>> The problem is where to put it if the virtio-ccw transport is used. The
> >>> idea was to put it in its own region (a device-memory region) and allow
> >>> the guest some way to discover that region (which is not within the
> >>> normal system memory from the guest's POW, just as pci memory isn't).
> >>> However, if we do that, we end up having the shared regions in
> >>> different regions depending upon whether the pci or the ccw transport
> >>> is used. Having pci use the special device memory on s390x does not
> >>> really sound like a good idea, either. Nor does putting it into pci
> >>> memory and accessing it from the ccw transport...  
> >>
> >> How should virtio_ccw access this memory? Only via CCW programs? In this
> >> case you don't have to put it in the guests address space. Or should
> >> standard memory instructions (lg,st) be used? In this case maybe the
> >> 1-to-1 mapping or the vmalloc space??
> > 
> > The idea was to use standard memory instructions (i.e. both host and
> > guest can read from/write to the region at any time). There was also
> > the idea to read/write this from the guest side only via ccws, for
> > which we wouldn't need a new address space, but that would have serious
> > drawbacks like not being able to read/write from an atomic context.
> > 
> 
> Or mapping into into user space, if that would ever be required.

The specific case this discussion started for was explicitly allowing
it to be mapped all the way to user space;  in virtio-fs we're mapping
normal files into this space in the qemu and they're presented as DAX
inside the guest kernel and eventually mapped through to the user
process.

Dave

> -- 
> 
> Thanks,
> 
> David / dhildenb
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Mapping memory regions on s390

[Cornelia Huck]

On Fri, 1 Mar 2019 14:50:14 +0100
David Hildenbrand <david@redhat.com> wrote:

> On 01.03.19 14:45, Cornelia Huck wrote:
> > On Fri, 1 Mar 2019 14:18:21 +0100 (CET)
> > Sebastian Ott <sebott@linux.ibm.com> wrote:
> >   
> >> On Fri, 1 Mar 2019, Cornelia Huck wrote:  
> >>> On Fri, 1 Mar 2019 13:14:38 +0100 (CET)
> >>> Sebastian Ott <sebott@linux.ibm.com> wrote:    
> >   
> >>>> The iomap, readb stuff is only usable in a PCI context on s390. But what
> >>>> is the problem here? virtio_ccw knows it's not a PCI driver - it doesn't
> >>>> have to use this stuff..    
> >>>
> >>> The device has to put the shared regions somewhere. If the virtio-pci
> >>> transport is used, it's just a normal area within a BAR (IIUC), and
> >>> should be handled just like normal pci memory.
> >>>
> >>> The problem is where to put it if the virtio-ccw transport is used. The
> >>> idea was to put it in its own region (a device-memory region) and allow
> >>> the guest some way to discover that region (which is not within the
> >>> normal system memory from the guest's POW, just as pci memory isn't).
> >>> However, if we do that, we end up having the shared regions in
> >>> different regions depending upon whether the pci or the ccw transport
> >>> is used. Having pci use the special device memory on s390x does not
> >>> really sound like a good idea, either. Nor does putting it into pci
> >>> memory and accessing it from the ccw transport...    
> >>
> >> How should virtio_ccw access this memory? Only via CCW programs? In this
> >> case you don't have to put it in the guests address space. Or should
> >> standard memory instructions (lg,st) be used? In this case maybe the
> >> 1-to-1 mapping or the vmalloc space??  
> > 
> > The idea was to use standard memory instructions (i.e. both host and
> > guest can read from/write to the region at any time). There was also
> > the idea to read/write this from the guest side only via ccws, for
> > which we wouldn't need a new address space, but that would have serious
> > drawbacks like not being able to read/write from an atomic context.
> >   
> 
> Or mapping into into user space, if that would ever be required.
> 

Sorry about the thread necromancy here, but I'm still not quite sure
how to proceed.

My understanding is that we can't use any other way to access this
memory than normal read/write, or it would clash with at least one of
the main use cases for this.

This leaves the problem that we support both virtio-ccw and virtio-pci
devices on s390 guests. The three options I see are:

(1) use a special device memory area for ccw and normal pci memory for
pci

(2) have pci use the special device memory area as well

(3) have ccw use pci memory

I don't really like any of these... (2) would make pci different on
s390 than anywhere else (well, even more so than it already is), and
(3) feels wrong as well. (1) is probably most benign in practice, but
would likely require us to configure devices differently depending on
the transport used, which seems troublesome as well.

I also have no idea if the protected guest thing will bite us here in
any way, especially as I have no access to the architecture.

Thoughts? Am I missing a fourth solution, or will we have to pick the
least bad one?