about QEMU TLS

about QEMU TLS

[Yu Zhang]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

Hi all,

According to qemu docs [1], TLS parameters are specified as an object in
the QEMU command line:

   -object tls-creds-x509,id=id,endpoint=endpoint,dir=/path/to/cred/dir ...

of which "endpoint" is a type of "QCryptoTLSCredsEndpoint" and can be
either a "server" or a "client".

I'd like to know:

- When a VM is started with this config, is there a way (e.g. QMP) to
change the value of "endpoint"?
  If possible, how to do this? or else after the first migration of a VM,
the VM has "endpoint=server",
  which can't be migrated without stop / start.

- In which case does the QEMU reload its TLS certificate, e.g. when a QEMU
VM has been run longer
  than the valid period of its TLS certificate?

- The migration is done by using HMP monitor on both source and target
side. Is it possible to do it
  by using QMP commands?

[1] https://www.qemu.org/docs/master/system/tls.html
[2]
https://www.berrange.com/posts/2016/08/16/improving-qemu-security-part-7-tls-support-for-migration/

Thank you so much for your reply!

Yu Zhang @ Compute Platform IONOS
06.08.2023

[-- Attachment #2: Type: text/html, Size: 1478 bytes --]

Re: about QEMU TLS

[Daniel P. Berrangé]

On Mon, Aug 07, 2023 at 12:07:31AM +0200, Yu Zhang wrote:
> Hi all,
> 
> According to qemu docs [1], TLS parameters are specified as an object in
> the QEMU command line:
> 
>    -object tls-creds-x509,id=id,endpoint=endpoint,dir=/path/to/cred/dir ...
> 
> of which "endpoint" is a type of "QCryptoTLSCredsEndpoint" and can be
> either a "server" or a "client".
> 
> I'd like to know:
> 
> - When a VM is started with this config, is there a way (e.g. QMP) to
> change the value of "endpoint"?
>   If possible, how to do this? or else after the first migration of a VM,
> the VM has "endpoint=server",
>   which can't be migrated without stop / start.

Use object_del + object_add to delete the old credentials and
create new ones.

> - In which case does the QEMU reload its TLS certificate, e.g. when a QEMU
> VM has been run longer
>   than the valid period of its TLS certificate?

The certs are loaded at the time the incoming/outgoing migration
operation is initiated, so they are always fresh.

> - The migration is done by using HMP monitor on both source and target
> side. Is it possible to do it
>   by using QMP commands?

Almost everything in HMP has an equivalent QMP command.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: about QEMU TLS

[Yu Zhang]

Hello Daniel,

sorry for my slow reply! I tested the approach you suggested by the
following way:

On the target server, start a VM in -incoming mode:

qemu-7.1 \
-uuid ${VM_UUID} \
 ...
-object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=server \
 ...
-incoming defer \
-qmp unix:${SOCK},server,nowait \
-qmp unix:${SOCK},server,nowait &

Set the migrate parameter and waiting for the incoming VM from source:

echo '{"execute":"qmp_capabilities"}{ "execute":
"migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
sudo nc -U -w 1 ${SOCK}
echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
"arguments": { "uri": "tcp::8089" }}

in HMP:
(qemu) migrate_set_parameter tls-creds tls0
(qemu) migrate_incoming tcp:[::]:8089

On the source server, start a VM:

qemu-7.1 \
-uuid ${VM_UUID} \
 ...
-object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client \
 ...
-qmp unix:${SOCK},server,nowait \
-qmp unix:${SOCK},server,nowait &

Set the migrate parameter and migrate the VM from source to target:

echo '{"execute":"qmp_capabilities"}{ "execute":
"migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
sudo nc -U -w 1 ${SOCK}
echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
"arguments": { "uri": "tcp:10.41.19.32:8089" }}

and query the migration after a few seconds:

echo '{"execute":"qmp_capabilities"}{ "execute": "query-migrate" }' |
sudo nc -U -w 1 ${SOCK}

the migrate is completed successfully.

To further migrate the VM from source (the target for the previously
migration), the endpoint must be changed from "server" to "client" by
QMP commands:

echo '{"execute":"qmp_capabilities"}{ "execute": "object-del",
"arguments": { "id": "tls0" }}' | sudo nc -U -w 1 ${SOCK}
echo '{"execute":"qmp_capabilities"}{ "execute": "object-add",
"arguments": { "id": "tls0", "qom-type": "tls-creds-x509", "endpoint":
"client", "dir": "${HOME}/qemutls", "verify-peer": false }}' | sudo nc
-U -w 1 ${SOCK}

which in HMP commands are:

(qemu) object_del tls0
(qemu) object_add tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client
(qemu) migrate_set_parameter tls-creds tls0
(qemu) migrate tcp:10.41.16.10:8089

So far as I tested, the TLS certificate must be valid for at least one
day. Therefore, the VM migration with an expired TLS certificate can
only be done in one day.

Thank you so much for your kind reply!
Best regards

Yu Zhang @ IONOS Compute Platform

On Thu, Aug 17, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Aug 07, 2023 at 12:07:31AM +0200, Yu Zhang wrote:
> > Hi all,
> >
> > According to qemu docs [1], TLS parameters are specified as an object in
> > the QEMU command line:
> >
> >    -object tls-creds-x509,id=id,endpoint=endpoint,dir=/path/to/cred/dir ...
> >
> > of which "endpoint" is a type of "QCryptoTLSCredsEndpoint" and can be
> > either a "server" or a "client".
> >
> > I'd like to know:
> >
> > - When a VM is started with this config, is there a way (e.g. QMP) to
> > change the value of "endpoint"?
> >   If possible, how to do this? or else after the first migration of a VM,
> > the VM has "endpoint=server",
> >   which can't be migrated without stop / start.
>
> Use object_del + object_add to delete the old credentials and
> create new ones.
>
> > - In which case does the QEMU reload its TLS certificate, e.g. when a QEMU
> > VM has been run longer
> >   than the valid period of its TLS certificate?
>
> The certs are loaded at the time the incoming/outgoing migration
> operation is initiated, so they are always fresh.
>
> > - The migration is done by using HMP monitor on both source and target
> > side. Is it possible to do it
> >   by using QMP commands?
>
> Almost everything in HMP has an equivalent QMP command.
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: about QEMU TLS

[Yu Zhang]

Hello Daniel and all,

When I was using TLS encryption for VM live-migration, I noticed one
thing: the migration works regardless of the "endpoint" setting (that
is: either "endpoint=server", or "endpoint=client") on the target
server.
The line I added is:
"-object tls-creds-x509,id=tls0,dir=/path/to/qemutls,endpoint=client
(or server),verify-peer=on".

It seems that currently the setting of "endpoint" is not strictly
enforced for VM migration. I'd like to know, if it's intentionally
done to allow a certain flexibility, or should be fixed from the
security perspective. Thank you very much!

Best regards,
Yu Zhang @ IONOS cloud

On Mon, Aug 21, 2023 at 4:29 PM Yu Zhang <yu.zhang@ionos.com> wrote:
>
> Hello Daniel,
>
> sorry for my slow reply! I tested the approach you suggested by the
> following way:
>
> On the target server, start a VM in -incoming mode:
>
> qemu-7.1 \
> -uuid ${VM_UUID} \
>  ...
> -object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=server \
>  ...
> -incoming defer \
> -qmp unix:${SOCK},server,nowait \
> -qmp unix:${SOCK},server,nowait &
>
> Set the migrate parameter and waiting for the incoming VM from source:
>
> echo '{"execute":"qmp_capabilities"}{ "execute":
> "migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
> sudo nc -U -w 1 ${SOCK}
> echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
> "arguments": { "uri": "tcp::8089" }}
>
> in HMP:
> (qemu) migrate_set_parameter tls-creds tls0
> (qemu) migrate_incoming tcp:[::]:8089
>
> On the source server, start a VM:
>
> qemu-7.1 \
> -uuid ${VM_UUID} \
>  ...
> -object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client \
>  ...
> -qmp unix:${SOCK},server,nowait \
> -qmp unix:${SOCK},server,nowait &
>
> Set the migrate parameter and migrate the VM from source to target:
>
> echo '{"execute":"qmp_capabilities"}{ "execute":
> "migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
> sudo nc -U -w 1 ${SOCK}
> echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
> "arguments": { "uri": "tcp:10.41.19.32:8089" }}
>
> and query the migration after a few seconds:
>
> echo '{"execute":"qmp_capabilities"}{ "execute": "query-migrate" }' |
> sudo nc -U -w 1 ${SOCK}
>
> the migrate is completed successfully.
>
> To further migrate the VM from source (the target for the previously
> migration), the endpoint must be changed from "server" to "client" by
> QMP commands:
>
> echo '{"execute":"qmp_capabilities"}{ "execute": "object-del",
> "arguments": { "id": "tls0" }}' | sudo nc -U -w 1 ${SOCK}
> echo '{"execute":"qmp_capabilities"}{ "execute": "object-add",
> "arguments": { "id": "tls0", "qom-type": "tls-creds-x509", "endpoint":
> "client", "dir": "${HOME}/qemutls", "verify-peer": false }}' | sudo nc
> -U -w 1 ${SOCK}
>
> which in HMP commands are:
>
> (qemu) object_del tls0
> (qemu) object_add tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client
> (qemu) migrate_set_parameter tls-creds tls0
> (qemu) migrate tcp:10.41.16.10:8089
>
> So far as I tested, the TLS certificate must be valid for at least one
> day. Therefore, the VM migration with an expired TLS certificate can
> only be done in one day.
>
> Thank you so much for your kind reply!
> Best regards
>
> Yu Zhang @ IONOS Compute Platform
>
> On Thu, Aug 17, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Mon, Aug 07, 2023 at 12:07:31AM +0200, Yu Zhang wrote:
> > > Hi all,
> > >
> > > According to qemu docs [1], TLS parameters are specified as an object in
> > > the QEMU command line:
> > >
> > >    -object tls-creds-x509,id=id,endpoint=endpoint,dir=/path/to/cred/dir ...
> > >
> > > of which "endpoint" is a type of "QCryptoTLSCredsEndpoint" and can be
> > > either a "server" or a "client".
> > >
> > > I'd like to know:
> > >
> > > - When a VM is started with this config, is there a way (e.g. QMP) to
> > > change the value of "endpoint"?
> > >   If possible, how to do this? or else after the first migration of a VM,
> > > the VM has "endpoint=server",
> > >   which can't be migrated without stop / start.
> >
> > Use object_del + object_add to delete the old credentials and
> > create new ones.
> >
> > > - In which case does the QEMU reload its TLS certificate, e.g. when a QEMU
> > > VM has been run longer
> > >   than the valid period of its TLS certificate?
> >
> > The certs are loaded at the time the incoming/outgoing migration
> > operation is initiated, so they are always fresh.
> >
> > > - The migration is done by using HMP monitor on both source and target
> > > side. Is it possible to do it
> > >   by using QMP commands?
> >
> > Almost everything in HMP has an equivalent QMP command.
> >
> >
> > With regards,
> > Daniel
> > --
> > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> >

Re: about QEMU TLS

[Yu Zhang]

Sorry for my confusion. I tested TLS migration by using RDMA, as RDMA
traffic bypasses the CPU, the TLS setting is not validated. With TCP,
the connection can't be established if "endpoint" setting is wrong.

On Tue, Jun 11, 2024 at 5:57 PM Yu Zhang <yu.zhang@ionos.com> wrote:
>
> Hello Daniel and all,
>
> When I was using TLS encryption for VM live-migration, I noticed one
> thing: the migration works regardless of the "endpoint" setting (that
> is: either "endpoint=server", or "endpoint=client") on the target
> server.
> The line I added is:
> "-object tls-creds-x509,id=tls0,dir=/path/to/qemutls,endpoint=client
> (or server),verify-peer=on".
>
> It seems that currently the setting of "endpoint" is not strictly
> enforced for VM migration. I'd like to know, if it's intentionally
> done to allow a certain flexibility, or should be fixed from the
> security perspective. Thank you very much!
>
> Best regards,
> Yu Zhang @ IONOS cloud
>
> On Mon, Aug 21, 2023 at 4:29 PM Yu Zhang <yu.zhang@ionos.com> wrote:
> >
> > Hello Daniel,
> >
> > sorry for my slow reply! I tested the approach you suggested by the
> > following way:
> >
> > On the target server, start a VM in -incoming mode:
> >
> > qemu-7.1 \
> > -uuid ${VM_UUID} \
> >  ...
> > -object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=server \
> >  ...
> > -incoming defer \
> > -qmp unix:${SOCK},server,nowait \
> > -qmp unix:${SOCK},server,nowait &
> >
> > Set the migrate parameter and waiting for the incoming VM from source:
> >
> > echo '{"execute":"qmp_capabilities"}{ "execute":
> > "migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
> > sudo nc -U -w 1 ${SOCK}
> > echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
> > "arguments": { "uri": "tcp::8089" }}
> >
> > in HMP:
> > (qemu) migrate_set_parameter tls-creds tls0
> > (qemu) migrate_incoming tcp:[::]:8089
> >
> > On the source server, start a VM:
> >
> > qemu-7.1 \
> > -uuid ${VM_UUID} \
> >  ...
> > -object tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client \
> >  ...
> > -qmp unix:${SOCK},server,nowait \
> > -qmp unix:${SOCK},server,nowait &
> >
> > Set the migrate parameter and migrate the VM from source to target:
> >
> > echo '{"execute":"qmp_capabilities"}{ "execute":
> > "migrate-set-parameters", "arguments": { "tls-creds": "tls0" }}' |
> > sudo nc -U -w 1 ${SOCK}
> > echo '{"execute":"qmp_capabilities"}{ "execute": "migrate",
> > "arguments": { "uri": "tcp:10.41.19.32:8089" }}
> >
> > and query the migration after a few seconds:
> >
> > echo '{"execute":"qmp_capabilities"}{ "execute": "query-migrate" }' |
> > sudo nc -U -w 1 ${SOCK}
> >
> > the migrate is completed successfully.
> >
> > To further migrate the VM from source (the target for the previously
> > migration), the endpoint must be changed from "server" to "client" by
> > QMP commands:
> >
> > echo '{"execute":"qmp_capabilities"}{ "execute": "object-del",
> > "arguments": { "id": "tls0" }}' | sudo nc -U -w 1 ${SOCK}
> > echo '{"execute":"qmp_capabilities"}{ "execute": "object-add",
> > "arguments": { "id": "tls0", "qom-type": "tls-creds-x509", "endpoint":
> > "client", "dir": "${HOME}/qemutls", "verify-peer": false }}' | sudo nc
> > -U -w 1 ${SOCK}
> >
> > which in HMP commands are:
> >
> > (qemu) object_del tls0
> > (qemu) object_add tls-creds-x509,id=tls0,dir=${HOME}/qemutls,endpoint=client
> > (qemu) migrate_set_parameter tls-creds tls0
> > (qemu) migrate tcp:10.41.16.10:8089
> >
> > So far as I tested, the TLS certificate must be valid for at least one
> > day. Therefore, the VM migration with an expired TLS certificate can
> > only be done in one day.
> >
> > Thank you so much for your kind reply!
> > Best regards
> >
> > Yu Zhang @ IONOS Compute Platform
> >
> > On Thu, Aug 17, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Mon, Aug 07, 2023 at 12:07:31AM +0200, Yu Zhang wrote:
> > > > Hi all,
> > > >
> > > > According to qemu docs [1], TLS parameters are specified as an object in
> > > > the QEMU command line:
> > > >
> > > >    -object tls-creds-x509,id=id,endpoint=endpoint,dir=/path/to/cred/dir ...
> > > >
> > > > of which "endpoint" is a type of "QCryptoTLSCredsEndpoint" and can be
> > > > either a "server" or a "client".
> > > >
> > > > I'd like to know:
> > > >
> > > > - When a VM is started with this config, is there a way (e.g. QMP) to
> > > > change the value of "endpoint"?
> > > >   If possible, how to do this? or else after the first migration of a VM,
> > > > the VM has "endpoint=server",
> > > >   which can't be migrated without stop / start.
> > >
> > > Use object_del + object_add to delete the old credentials and
> > > create new ones.
> > >
> > > > - In which case does the QEMU reload its TLS certificate, e.g. when a QEMU
> > > > VM has been run longer
> > > >   than the valid period of its TLS certificate?
> > >
> > > The certs are loaded at the time the incoming/outgoing migration
> > > operation is initiated, so they are always fresh.
> > >
> > > > - The migration is done by using HMP monitor on both source and target
> > > > side. Is it possible to do it
> > > >   by using QMP commands?
> > >
> > > Almost everything in HMP has an equivalent QMP command.
> > >
> > >
> > > With regards,
> > > Daniel
> > > --
> > > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> > >