* vsock CID questions @ 2020-02-18 22:45 ted.h.kim 2020-02-19 10:44 ` Stefano Garzarella 2020-02-19 15:43 ` Stefan Hajnoczi 0 siblings, 2 replies; 6+ messages in thread From: ted.h.kim @ 2020-02-18 22:45 UTC (permalink / raw) To: sgarzare; +Cc: stefanha, netdev Hi Stefano (and Stefan), I have some questions about vsock CIDs, particularly when migration happens. 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)? 2. In the vsock(7) man page, it says the CID might change upon migration, if it is not available. Is there some notification when CID reassignment happens? 3. if CID reassignment happens, is this persistent? (i.e. will I see updated vsock definition in XML for the guest) 4. I would like to minimize the chance of CID collision. If I understand correctly, the CID is a 32-bit unsigned. So for my application, it might work to put an IPv4 address. But if I adopt this convention, then I need to look forward to possibly using IPv6. Anyway, would it be hard to potentially expand the size of the CID to 64 bits or even 128? Thanks, -ted -- Ted H. Kim, PhD ted.h.kim@oracle.com +1 310-258-7515 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: vsock CID questions 2020-02-18 22:45 vsock CID questions ted.h.kim @ 2020-02-19 10:44 ` Stefano Garzarella 2020-02-19 15:43 ` Stefan Hajnoczi 1 sibling, 0 replies; 6+ messages in thread From: Stefano Garzarella @ 2020-02-19 10:44 UTC (permalink / raw) To: ted.h.kim; +Cc: stefanha, netdev On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote: > Hi Stefano (and Stefan), Hi Ted, > > I have some questions about vsock CIDs, particularly when migration happens. > > 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)? I don't know if there is a specific API, but looking at the xml, you can see the assigned CID: $ virsh dumpxml fedora31 | grep cid <cid auto='yes' address='3'/> I'm not sure that's what you were asking, if you meant a list of all the guest CIDs, I don't think there's an API for that. > > 2. In the vsock(7) man page, it says the CID might change upon migration, if > it is not available. > Is there some notification when CID reassignment happens? Connected stream sockets will receive an error after the migration and then they'll be closed. Usually it is not recommended to bind the guest's cid, it is preferable to use VMADDR_CID_ANY. > > 3. if CID reassignment happens, is this persistent? (i.e. will I see updated > vsock definition in XML for the guest) I guess so, but I didn't try. > > 4. I would like to minimize the chance of CID collision. If I understand > correctly, the CID is a 32-bit unsigned. Right. 'struct sockaddr_vm' supports 32-bit unsigned CID. > So for my application, it might > work to put an IPv4 address. But if I adopt this convention, then I need to > look forward to possibly using IPv6. Anyway, would it be hard to potentially > expand the size of the CID to 64 bits or even 128? virtio-vsock specification [1] supports up to 64-bit CID. The 'svm_cid' field in the 'struct sockaddr_vm' is the last one, before the zero section, and we have 16-bit reserved on top that we can use for some flags. Maybe extending it to 64 bit might be feasible, but we need to check other transports (vmci, hyperv). Cheers, Stefano [1] https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-3960006 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: vsock CID questions 2020-02-18 22:45 vsock CID questions ted.h.kim 2020-02-19 10:44 ` Stefano Garzarella @ 2020-02-19 15:43 ` Stefan Hajnoczi 2020-02-20 16:09 ` Ján Tomko 1 sibling, 1 reply; 6+ messages in thread From: Stefan Hajnoczi @ 2020-02-19 15:43 UTC (permalink / raw) To: ted.h.kim; +Cc: sgarzare, netdev, Ján Tomko [-- Attachment #1: Type: text/plain, Size: 1574 bytes --] On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote: > 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)? I wonder if it can be queried from libvirt (at a minimum the domain XML might have the CID)? I have CCed Ján Tomko who worked on the libvirt support: https://libvirt.org/formatdomain.html#vsock > 2. In the vsock(7) man page, it says the CID might change upon migration, if > it is not available. > Is there some notification when CID reassignment happens? All established connections are reset across live migration - applications will notice :). Listen sockets stay open but automatically listen on the new CID. > 3. if CID reassignment happens, is this persistent? (i.e. will I see updated > vsock definition in XML for the guest) Another question for Ján. > 4. I would like to minimize the chance of CID collision. If I understand > correctly, the CID is a 32-bit unsigned. So for my application, it might > work to put an IPv4 address. But if I adopt this convention, then I need to > look forward to possibly using IPv6. Anyway, would it be hard to potentially > expand the size of the CID to 64 bits or even 128? A little hard, since the struct sockaddr_vm that userspace applications use has a 32-bit CID field. This is because the existing VMware VMCI vsock implementation has 32-bit CIDs. virtio-vsock is ready for 64-bit CIDs (the packet header fields are already 64-bit) but changes to net/vmw_vsock/ core code and to the userspace ABI would be necessary. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: vsock CID questions 2020-02-19 15:43 ` Stefan Hajnoczi @ 2020-02-20 16:09 ` Ján Tomko 2020-02-21 19:49 ` ted.h.kim 0 siblings, 1 reply; 6+ messages in thread From: Ján Tomko @ 2020-02-20 16:09 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: ted.h.kim, sgarzare, netdev [-- Attachment #1: Type: text/plain, Size: 2351 bytes --] On Wed, Feb 19, 2020 at 03:43:17PM +0000, Stefan Hajnoczi wrote: >On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote: >> 1. Is there an API to lookup CIDs of guests from the host side (in libvirt)? > >I wonder if it can be queried from libvirt (at a minimum the domain XML >might have the CID)? I have CCed Ján Tomko who worked on the libvirt >support: > >https://libvirt.org/formatdomain.html#vsock > Yes, libvirt has to know the CIDs of the guest and presents them in the domain XML: <domain type='kvm'> <name>test</name> ... <devices> ... <vsock model='virtio'> <cid auto='no' address='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </vsock> </devices> </domain> >> 2. In the vsock(7) man page, it says the CID might change upon migration, if >> it is not available. >> Is there some notification when CID reassignment happens? > >All established connections are reset across live migration - >applications will notice :). > >Listen sockets stay open but automatically listen on the new CID. > >> 3. if CID reassignment happens, is this persistent? (i.e. will I see updated >> vsock definition in XML for the guest) > >Another question for Ján. Depends on the setting. For <cid auto='yes'/>, libvirt will try to acquire the first available CID for the guest and pass it to QEMU. For <cid auto='no'/>, no reassignment should happend and the CID requested in the domain XML on the source will be used (or fail to be used) on the destination. Jano > >> 4. I would like to minimize the chance of CID collision. If I understand >> correctly, the CID is a 32-bit unsigned. So for my application, it might >> work to put an IPv4 address. But if I adopt this convention, then I need to >> look forward to possibly using IPv6. Anyway, would it be hard to potentially >> expand the size of the CID to 64 bits or even 128? > >A little hard, since the struct sockaddr_vm that userspace applications >use has a 32-bit CID field. This is because the existing VMware VMCI >vsock implementation has 32-bit CIDs. > >virtio-vsock is ready for 64-bit CIDs (the packet header fields are >already 64-bit) but changes to net/vmw_vsock/ core code and to the >userspace ABI would be necessary. > >Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: vsock CID questions 2020-02-20 16:09 ` Ján Tomko @ 2020-02-21 19:49 ` ted.h.kim 2020-02-25 11:30 ` Ján Tomko 0 siblings, 1 reply; 6+ messages in thread From: ted.h.kim @ 2020-02-21 19:49 UTC (permalink / raw) To: Ján Tomko, Stefan Hajnoczi; +Cc: sgarzare, netdev, Boris Ostrovsky Hi Jan, Thanks for responding - let me see if I am understanding correctly. I think you are saying on migration the process of determining the CID assigned is the same as when you start a domain. <cid auto='yes'> means assignment to the first available value. It also seems for auto='yes' that any address='<value>' part of the CID definition is ignored, even as a suggested value. (I always get CID 3 when starting auto='yes' and no other domains have started, even if there is an specific address in the definition, e.g. address='12'.) But if auto='no', either the domain gets the address field value or if the value is already assigned, the domain will fail to start/migrate. Is that right? In cases, where auto='yes', it does not seem that the host/hypervisor can find out what CID value was assigned to a domain. Even parsing the XML only reveals that it was auto-assigned and no specific value can be determined. Is this correct? If this is the case, I would advocate for a specific API which can lookup the current CID of a domain. Otherwise the host/hypervisor cannot tell which auto='yes' domain is on the other end of the connected socket, when there is more than one. Thanks. -ted On 2/20/20 8:09 AM, Ján Tomko wrote: > On Wed, Feb 19, 2020 at 03:43:17PM +0000, Stefan Hajnoczi wrote: >> On Tue, Feb 18, 2020 at 02:45:38PM -0800, ted.h.kim@oracle.com wrote: >>> 1. Is there an API to lookup CIDs of guests from the host side (in >>> libvirt)? >> >> I wonder if it can be queried from libvirt (at a minimum the domain XML >> might have the CID)? I have CCed Ján Tomko who worked on the libvirt >> support: >> >> https://libvirt.org/formatdomain.html#vsock >> > > Yes, libvirt has to know the CIDs of the guest and presents them in the > domain XML: > <domain type='kvm'> > <name>test</name> > ... > <devices> > ... > <vsock model='virtio'> > <cid auto='no' address='4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > function='0x0'/> > </vsock> > </devices> > </domain> > >>> 2. In the vsock(7) man page, it says the CID might change upon >>> migration, if >>> it is not available. >>> Is there some notification when CID reassignment happens? >> >> All established connections are reset across live migration - >> applications will notice :). >> >> Listen sockets stay open but automatically listen on the new CID. >> >>> 3. if CID reassignment happens, is this persistent? (i.e. will I see >>> updated >>> vsock definition in XML for the guest) >> >> Another question for Ján. > > Depends on the setting. > For <cid auto='yes'/>, libvirt will try to acquire the first available > CID > for the guest and pass it to QEMU. > For <cid auto='no'/>, no reassignment should happend and the CID > requested in the domain XML on the source will be used (or fail to be > used) on the destination. > > Jano > >> >>> 4. I would like to minimize the chance of CID collision. If I >>> understand >>> correctly, the CID is a 32-bit unsigned. So for my application, it >>> might >>> work to put an IPv4 address. But if I adopt this convention, then I >>> need to >>> look forward to possibly using IPv6. Anyway, would it be hard to >>> potentially >>> expand the size of the CID to 64 bits or even 128? >> >> A little hard, since the struct sockaddr_vm that userspace applications >> use has a 32-bit CID field. This is because the existing VMware VMCI >> vsock implementation has 32-bit CIDs. >> >> virtio-vsock is ready for 64-bit CIDs (the packet header fields are >> already 64-bit) but changes to net/vmw_vsock/ core code and to the >> userspace ABI would be necessary. >> >> Stefan > > -- Ted H. Kim, PhD ted.h.kim@oracle.com +1 310-258-7515 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: vsock CID questions 2020-02-21 19:49 ` ted.h.kim @ 2020-02-25 11:30 ` Ján Tomko 0 siblings, 0 replies; 6+ messages in thread From: Ján Tomko @ 2020-02-25 11:30 UTC (permalink / raw) To: ted.h.kim; +Cc: Stefan Hajnoczi, sgarzare, netdev, Boris Ostrovsky [-- Attachment #1: Type: text/plain, Size: 1855 bytes --] On Fri, Feb 21, 2020 at 11:49:06AM -0800, ted.h.kim@oracle.com wrote: >Hi Jan, > >Thanks for responding - let me see if I am understanding correctly. > > >I think you are saying on migration the process of determining the CID >assigned is the same as when you start a domain. > ><cid auto='yes'> means assignment to the first available value. >It also seems for auto='yes' that any address='<value>' part of the >CID definition is ignored, even as a suggested value. >(I always get CID 3 when starting auto='yes' and no other domains have >started, even if there is an specific address in the definition, e.g. >address='12'.) > >But if auto='no', either the domain gets the address field value or if >the value is already assigned, the domain will fail to start/migrate. > >Is that right? > Yes. > >In cases, where auto='yes', it does not seem that the host/hypervisor >can find out what CID value was assigned to a domain. Even parsing the >XML only reveals that it was auto-assigned and no specific value can >be determined. The value is not recorded in the inactive XML, but after domain startup, the address should be visible in the live XML: <vsock model="virtio"> <cid auto="yes" address="3"/> <alias name="vsock0"/> <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/> </vsock> > >Is this correct? > >If this is the case, I would advocate for a specific API which can >lookup the current CID of a domain. This can already be queried from the XML through virDomainGetXMLDesc: $ virsh dumpxml libvirt-fedora-31 | xmllint --xpath 'string(//cid/@address)' - 3 Jano >Otherwise the host/hypervisor cannot tell which auto='yes' domain is >on the other end of the connected socket, when there is more than one. > > >Thanks. >-ted > > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-02-25 11:30 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-02-18 22:45 vsock CID questions ted.h.kim 2020-02-19 10:44 ` Stefano Garzarella 2020-02-19 15:43 ` Stefan Hajnoczi 2020-02-20 16:09 ` Ján Tomko 2020-02-21 19:49 ` ted.h.kim 2020-02-25 11:30 ` Ján Tomko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).