Hotplugged devices in Xen 4.5 and domain reboot

Hotplugged devices in Xen 4.5 and domain reboot

[Pavlo Suikov]

[-- Attachment #1.1: Type: text/plain, Size: 1293 bytes --]

Hi,

got a question regarding hotplugged devices in Xen 4.5. What we currently
have is Xen 4.5, Linux in Dom0 and DomD and Android in DomU. All the
devices are configured via domain config, so they are initialized on domain
startup: libxl reads config, sets up xenstore branches and fills them for
driver back ends and front ends; front ends monitor these branches and fill
in their specific data once connection is established. On DomU reboot,
config is being re-read and all the devices are started again in the same
manner.

Problem is we don't really know what to do with hotplugged devices. When
such a device appears in DomU during runtime, we set up all the xenstore
insformation on the fly, but it doesn't go to persistent domain
configuration; so when DomU reboot comes, they are not set up again as they
should. All the solutions we are considering look like dirty hacks (e.g.
making separate branch in xenstore for hotplugged devices, reading it on
domain start in the same way domain config is parsed, and copying data to
corresponding xenstore domain branches). Are there any existing ways to
handle these?

Suikov Pavlo
GlobalLogic
P +x.xxx.xxx.xxxx  M +38.066.667.1296  S psujkov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 3355 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Wei Liu]

On Tue, Dec 01, 2015 at 03:24:13PM +0200, Pavlo Suikov wrote:
> Hi,
> 
> got a question regarding hotplugged devices in Xen 4.5. What we currently
> have is Xen 4.5, Linux in Dom0 and DomD and Android in DomU. All the
> devices are configured via domain config, so they are initialized on domain
> startup: libxl reads config, sets up xenstore branches and fills them for
> driver back ends and front ends; front ends monitor these branches and fill
> in their specific data once connection is established. On DomU reboot,
> config is being re-read and all the devices are started again in the same
> manner.
> 
> Problem is we don't really know what to do with hotplugged devices. When
> such a device appears in DomU during runtime, we set up all the xenstore
> insformation on the fly, but it doesn't go to persistent domain
> configuration; so when DomU reboot comes, they are not set up again as they
> should. All the solutions we are considering look like dirty hacks (e.g.
> making separate branch in xenstore for hotplugged devices, reading it on
> domain start in the same way domain config is parsed, and copying data to
> corresponding xenstore domain branches). Are there any existing ways to
> handle these?
> 

Do you have the exact runes you use to do the hotplug? How do you reboot
your DomU? What does /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json
look like before and after hotplug?

Wei.

> Suikov Pavlo
> GlobalLogic
> P +x.xxx.xxx.xxxx  M +38.066.667.1296  S psujkov
> www.globallogic.com
> <http://www.globallogic.com/>
> http://www.globallogic.com/email_disclaimer.txt

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Iurii Mykhalskyi]

[-- Attachment #1.1: Type: text/plain, Size: 2847 bytes --]

Our real usb mass-storage device are located at driver domain (DomD). So we
setup second block-device backend there.

To hotplug usb mass-storage from DomD we use follow command:

xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"

There was no support of attaching block-device in runtime from domain other
to Domain-0, so we have made some hacks to allow call block-attach command
from non-dom0 privileged domain.
One of patches was - don't update
/var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of this
command (because this log located on dom0 rootfs and we don't have any
access to it from DomD). So, there is no different in configs before and
after hotplug.

Did xen hot-plug mechanism relay on those config?



On Tue, Dec 1, 2015 at 4:02 PM, Wei Liu <wei.liu2@citrix.com> wrote:

> On Tue, Dec 01, 2015 at 03:24:13PM +0200, Pavlo Suikov wrote:
> > Hi,
> >
> > got a question regarding hotplugged devices in Xen 4.5. What we currently
> > have is Xen 4.5, Linux in Dom0 and DomD and Android in DomU. All the
> > devices are configured via domain config, so they are initialized on
> domain
> > startup: libxl reads config, sets up xenstore branches and fills them for
> > driver back ends and front ends; front ends monitor these branches and
> fill
> > in their specific data once connection is established. On DomU reboot,
> > config is being re-read and all the devices are started again in the same
> > manner.
> >
> > Problem is we don't really know what to do with hotplugged devices. When
> > such a device appears in DomU during runtime, we set up all the xenstore
> > insformation on the fly, but it doesn't go to persistent domain
> > configuration; so when DomU reboot comes, they are not set up again as
> they
> > should. All the solutions we are considering look like dirty hacks (e.g.
> > making separate branch in xenstore for hotplugged devices, reading it on
> > domain start in the same way domain config is parsed, and copying data to
> > corresponding xenstore domain branches). Are there any existing ways to
> > handle these?
> >
>
> Do you have the exact runes you use to do the hotplug? How do you reboot
> your DomU? What does /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json
> look like before and after hotplug?
>
> Wei.
>
> > Suikov Pavlo
> > GlobalLogic
> > P +x.xxx.xxx.xxxx  M +38.066.667.1296  S psujkov
> > www.globallogic.com
> > <http://www.globallogic.com/>
> > http://www.globallogic.com/email_disclaimer.txt
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
>


-- 

Iurii Mykhalskyi | Senior Software Engineer
GlobalLogic
P +38.044.492.9695x3664  M +38.096.311.5467  S mad-nemoi
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 6273 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Wei Liu]

On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
> Our real usb mass-storage device are located at driver domain (DomD). So we
> setup second block-device backend there.
> 
> To hotplug usb mass-storage from DomD we use follow command:
> 
> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> 

What happens if you run this in Dom0? I guess DomD doesn't respond to
the request?

> There was no support of attaching block-device in runtime from domain other
> to Domain-0, so we have made some hacks to allow call block-attach command
> from non-dom0 privileged domain.

So this is a special use case. This is the first time I know people
actually run xl block-attach in driver domain.

> One of patches was - don't update
> /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of this
> command (because this log located on dom0 rootfs and we don't have any
> access to it from DomD). So, there is no different in configs before and
> after hotplug.
> 

The state of $DOMID is recorded in libxl-json file. No wonder you lose
all state.

But even if you write those states, they are going to be inside driver
domain.  There is no way at the moment to synthesise the state inside
Dom0 and DomD into one. There is also difficulty in how you can split
the synthesised and dispatch the states to multiple entities again when
rebuilding a domain.

So I think having multiple entities managing state of one single domain
is bad. I think the proper way of making it work is to make hotplug
device from domain other than Dom0 work.

There is a daemon "xl devd" in driver domain. We might be able to teach
it to response to Dom0 toostack request. I'm a bit surprised if it
doesn't do that already. Did you forget to start that daemon?

Roger, Ian and Ian, any thought?

Wei.

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Ian Campbell]

On Tue, 2015-12-01 at 15:29 +0000, Wei Liu wrote:
> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
> > Our real usb mass-storage device are located at driver domain (DomD).
> > So we
> > setup second block-device backend there.
> > 
> > To hotplug usb mass-storage from DomD we use follow command:
> > 
> > xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> > 
> 
> What happens if you run this in Dom0? I guess DomD doesn't respond to
> the request?
> 
> > There was no support of attaching block-device in runtime from domain
> > other
> > to Domain-0, so we have made some hacks to allow call block-attach
> > command
> > from non-dom0 privileged domain.
> 
> So this is a special use case. This is the first time I know people
> actually run xl block-attach in driver domain.

Toolstack commands (xl *) should be run in the toolstack domain, not in the
driver domain.

I don't think it should be expected that the latter work (at least not
without a large amount of development work).

In general a driver domain would not be expected to have sufficient
privilege over e.g. a guest domain's /local/domain/domU/devices to create
the f.e. dirs.

> There is a daemon "xl devd" in driver domain. We might be able to teach
> it to response to Dom0 toostack request. I'm a bit surprised if it
> doesn't do that already. Did you forget to start that daemon?

That's the entire purpose of that daemon, isn't it?

> 
> Roger, Ian and Ian, any thought?
> 
> Wei.

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Roger Pau Monné]

Hello,

El 01/12/15 a les 15.58, Iurii Mykhalskyi ha escrit:
> Our real usb mass-storage device are located at driver domain (DomD). So we
> setup second block-device backend there.
> 
> To hotplug usb mass-storage from DomD we use follow command:
> 
> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"

This is not possible by design, you should only be able to execute `xl 
block-attach ...` from the control domain (Dom0). This is due to the 
fact that attaching a new device to a guest requires write permissions 
in the guest xenstore paths, which the driver domain should not have.

> 
> There was no support of attaching block-device in runtime from domain other
> to Domain-0, so we have made some hacks to allow call block-attach command
> from non-dom0 privileged domain.

Do you have either the `xl devd` command running or udev rules 
correctly setup inside of the driver domain?

Does something like the following work? If not, could you paste the 
error when running it with -vvv.

xl block-attach DomU format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev

Roger.

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Iurii Mykhalskyi]

[-- Attachment #1.1: Type: text/plain, Size: 6194 bytes --]

Thanks to all for a replays, please see my answers below:

On 12/01/2015 05:29 PM, Wei Liu wrote:
> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
>> Our real usb mass-storage device are located at driver domain (DomD). So we
>> setup second block-device backend there.
>>
>> To hotplug usb mass-storage from DomD we use follow command:
>>
>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
>>
> What happens if you run this in Dom0? I guess DomD doesn't respond to
> the request?
     Yes, there is no responded from domD, because actual storage device
are located there, and toolstack stuck on real device existence check.
>> There was no support of attaching block-device in runtime from domain other
>> to Domain-0, so we have made some hacks to allow call block-attach command
>> from non-dom0 privileged domain.
> So this is a special use case. This is the first time I know people
> actually run xl block-attach in driver domain.
Yes, this is special case and we this by our solution design.
>
>> One of patches was - don't update
>> /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of this
>> command (because this log located on dom0 rootfs and we don't have any
>> access to it from DomD). So, there is no different in configs before and
>> after hotplug.
>>
> The state of $DOMID is recorded in libxl-json file. No wonder you lose
> all state.
>
> But even if you write those states, they are going to be inside driver
> domain.  There is no way at the moment to synthesise the state inside
> Dom0 and DomD into one. There is also difficulty in how you can split
> the synthesised and dispatch the states to multiple entities again when
> rebuilding a domain.
>
> So I think having multiple entities managing state of one single domain
> is bad. I think the proper way of making it work is to make hotplug
> device from domain other than Dom0 work.
>
> There is a daemon "xl devd" in driver domain. We might be able to teach
> it to response to Dom0 toostack request. I'm a bit surprised if it
> doesn't do that already. Did you forget to start that daemon?
We can't run devd in driver domain, because it failed on connect to
xenstored socket (/var/run/xenstored/socket - we have xenstored running
only in dom0).
>
> Roger, Ian and Ian, any thought?
>
> Wei.

On 12/01/2015 05:41 PM, Ian Campbell wrote:
> On Tue, 2015-12-01 at 15:29 +0000, Wei Liu wrote:
>> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
>>> Our real usb mass-storage device are located at driver domain (DomD).
>>> So we
>>> setup second block-device backend there.
>>>
>>> To hotplug usb mass-storage from DomD we use follow command:
>>>
>>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
>>>
>> What happens if you run this in Dom0? I guess DomD doesn't respond to
>> the request?
>>
>>> There was no support of attaching block-device in runtime from domain
>>> other
>>> to Domain-0, so we have made some hacks to allow call block-attach
>>> command
>>> from non-dom0 privileged domain.
>> So this is a special use case. This is the first time I know people
>> actually run xl block-attach in driver domain.
> Toolstack commands (xl *) should be run in the toolstack domain, not in the
> driver domain.
>
> I don't think it should be expected that the latter work (at least not
> without a large amount of development work).
>
> In general a driver domain would not be expected to have sufficient
> privilege over e.g. a guest domain's /local/domain/domU/devices to create
> the f.e. dirs.
In our solution we have to create 2 full privileged domains - Dom0 and
DomD, so we need 2 toolstack domains.
    Any special privileges hack wasn't done - we need just to setup
additional permissions for DomD.
>> There is a daemon "xl devd" in driver domain. We might be able to teach
>> it to response to Dom0 toostack request. I'm a bit surprised if it
>> doesn't do that already. Did you forget to start that daemon?
> That's the entire purpose of that daemon, isn't it?
I can't find any valuable documentation or examples of use for this
daemon. Can you point me please to any documentation about it, please?
    Thank you.
>
>> Roger, Ian and Ian, any thought?
>>
>> Wei.

On 12/01/2015 05:56 PM, Roger Pau Monné wrote:
> Hello,
>
> El 01/12/15 a les 15.58, Iurii Mykhalskyi ha escrit:
>> Our real usb mass-storage device are located at driver domain (DomD). So we
>> setup second block-device backend there.
>>
>> To hotplug usb mass-storage from DomD we use follow command:
>>
>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> This is not possible by design, you should only be able to execute `xl
> block-attach ...` from the control domain (Dom0). This is due to the
> fact that attaching a new device to a guest requires write permissions
> in the guest xenstore paths, which the driver domain should not have.
As I mention earlier, we need this case by design of our solution. And
DomD in our most of same privilegies as Dom0,
    so access to xenstore isn't point of the problem.
>
>> There was no support of attaching block-device in runtime from domain other
>> to Domain-0, so we have made some hacks to allow call block-attach command
>> from non-dom0 privileged domain.
> Do you have either the `xl devd` command running or udev rules
> correctly setup inside of the driver domain?
Yes, we have setup our own udev rules, that executes "xl block-attach
.." during usb stick insert.
> Does something like the following work? If not, could you paste the
> error when running it with -vvv.
>
> xl block-attach DomU format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
In dom0 we have next issue:
/libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
/this issue occurs due to missing /dev/sda1 device (all hardware are
placed in DomD domain).

   In domD with our patches - all ok. /dev/sda1 successfully forwarded
to DomU.

>
> Roger.
>

With the best regards, Iurii.




[-- Attachment #1.2: Type: text/html, Size: 9155 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Ian Campbell]

On Tue, 2015-12-01 at 18:48 +0200, Iurii Mykhalskyi wrote:
> Thanks to all for a replays, please see my answers below:
> 
> On 12/01/2015 05:29 PM, Wei Liu wrote:
> > On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
> > > Our real usb mass-storage device are located at driver domain (DomD).
> > > So we
> > > setup second block-device backend there.
> > > 
> > > To hotplug usb mass-storage from DomD we use follow command:
> > > 
> > > xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> > > 
> > What happens if you run this in Dom0? I guess DomD doesn't respond to
> > the request?
>     Yes, there is no responded from domD, because actual storage device
> are located there, and toolstack stuck on real device existence check.

This check is a toolstack bug which should be fixed. We've squashed some of
them at various points, but I'd not be surprised if others remain.

> > > There was no support of attaching block-device in runtime from domain
> > > other
> > > to Domain-0, so we have made some hacks to allow call block-attach
> > > command
> > > from non-dom0 privileged domain.
> > So this is a special use case. This is the first time I know people
> > actually run xl block-attach in driver domain.
>     Yes, this is special case and we this by our solution design.
> > > One of patches was - don't update
> > > /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of
> > > this
> > > command (because this log located on dom0 rootfs and we don't have
> > > any
> > > access to it from DomD). So, there is no different in configs before
> > > and
> > > after hotplug.
> > > 
> > The state of $DOMID is recorded in libxl-json file. No wonder you lose
> > all state.
> > 
> > But even if you write those states, they are going to be inside driver
> > domain.  There is no way at the moment to synthesise the state inside
> > Dom0 and DomD into one. There is also difficulty in how you can split
> > the synthesised and dispatch the states to multiple entities again when
> > rebuilding a domain.
> > 
> > So I think having multiple entities managing state of one single domain
> > is bad. I think the proper way of making it work is to make hotplug
> > device from domain other than Dom0 work.
> > 
> > There is a daemon "xl devd" in driver domain. We might be able to teach
> > it to response to Dom0 toostack request. I'm a bit surprised if it
> > doesn't do that already. Did you forget to start that daemon?
>     We can't run devd in driver domain, because it failed on connect to
> xenstored socket (/var/run/xenstored/socket - we have xenstored running
> only in dom0).   

devd _should_ be able to talk to xenstored over the kernel provided
interface to the shared ring rather than the local socket.

It is certainly not expected that devd be colocated in the same domain as
happens to be running xenstored.

If this is not working then there is another bug.

> > In general a driver domain would not be expected to have sufficient
> > privilege over e.g. a guest domain's /local/domain/domU/devices to
> > create
> > the f.e. dirs.

>     In our solution we have to create 2 full privileged domains - Dom0 and DomD, so we need 2 toolstack domains.
>    Any special privileges hack wasn't done - we need just to setup additional permissions for DomD.

I'm afraid you simply cannot have 2 toolstack domains. The toolstack is a
singleton entity in a Xen system.

If you want to run toolstack operations from a non-toolstack domain then
you will need to arrange for some (likely out-of-band) mechanism for such
domains to ask the single toolstack domain to do something on their behalf.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Wei Liu]

On Tue, Dec 01, 2015 at 06:48:32PM +0200, Iurii Mykhalskyi wrote:
> On 12/01/2015 05:41 PM, Ian Campbell wrote:
> >On Tue, 2015-12-01 at 15:29 +0000, Wei Liu wrote:
> >>On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
> >>>Our real usb mass-storage device are located at driver domain (DomD).
> >>>So we
> >>>setup second block-device backend there.
> >>>
> >>>To hotplug usb mass-storage from DomD we use follow command:
> >>>
> >>>xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> >>>
> >>What happens if you run this in Dom0? I guess DomD doesn't respond to
> >>the request?
> >>
> >>>There was no support of attaching block-device in runtime from domain
> >>>other
> >>>to Domain-0, so we have made some hacks to allow call block-attach
> >>>command
> >>>from non-dom0 privileged domain.
> >>So this is a special use case. This is the first time I know people
> >>actually run xl block-attach in driver domain.
> >Toolstack commands (xl *) should be run in the toolstack domain, not in the
> >driver domain.
> >
> >I don't think it should be expected that the latter work (at least not
> >without a large amount of development work).
> >
> >In general a driver domain would not be expected to have sufficient
> >privilege over e.g. a guest domain's /local/domain/domU/devices to create
> >the f.e. dirs.
> In our solution we have to create 2 full privileged domains - Dom0 and
> DomD, so we need 2 toolstack domains.
>    Any special privileges hack wasn't done - we need just to setup
> additional permissions for DomD.
> >>There is a daemon "xl devd" in driver domain. We might be able to teach
> >>it to response to Dom0 toostack request. I'm a bit surprised if it
> >>doesn't do that already. Did you forget to start that daemon?
> >That's the entire purpose of that daemon, isn't it?
> I can't find any valuable documentation or examples of use for this
> daemon. Can you point me please to any documentation about it, please?
>    Thank you.

You just run "xl devd" in driver domain with a init script or systemd
unit.

To help debug, run xl -F devd

> >
> >>Roger, Ian and Ian, any thought?
> >>
> >>Wei.
> 
> On 12/01/2015 05:56 PM, Roger Pau Monné wrote:
> >Hello,
> >
> >El 01/12/15 a les 15.58, Iurii Mykhalskyi ha escrit:
> >>Our real usb mass-storage device are located at driver domain (DomD). So we
> >>setup second block-device backend there.
> >>
> >>To hotplug usb mass-storage from DomD we use follow command:
> >>
> >>xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
> >This is not possible by design, you should only be able to execute `xl
> >block-attach ...` from the control domain (Dom0). This is due to the
> >fact that attaching a new device to a guest requires write permissions
> >in the guest xenstore paths, which the driver domain should not have.
> As I mention earlier, we need this case by design of our solution. And
> DomD in our most of same privilegies as Dom0,
>    so access to xenstore isn't point of the problem.

It's a bit confusing because this email contains a) the design and b)
workaround in Xen to meet your design.

As I understand it, you have a driver domain (DomD), and you need to
attach the device inside DomD to DomU. At the moment Xen _should_ be
able to do that. There might be bugs, but we shall fix it. There are
bugs because not that many people use such setup.

Note that toolstack doesn't care if DomD being privileged or not.

> >>There was no support of attaching block-device in runtime from domain other
> >>to Domain-0, so we have made some hacks to allow call block-attach command
> >>from non-dom0 privileged domain.
> >Do you have either the `xl devd` command running or udev rules
> >correctly setup inside of the driver domain?
> Yes, we have setup our own udev rules, that executes "xl block-attach
> .." during usb stick insert.
> >Does something like the following work? If not, could you paste the
> >error when running it with -vvv.
> >
> >xl block-attach DomU format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
> In dom0 we have next issue:
> /libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
> vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
> /this issue occurs due to missing /dev/sda1 device (all hardware are
> placed in DomD domain).
> 

Looks like the path that is stat'ed is not correct.

Wei.

>   In domD with our patches - all ok. /dev/sda1 successfully forwarded
> to DomU.
> 
> >
> >Roger.
> >
> 
> With the best regards, Iurii.
> 
> 
> 

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Roger Pau Monné]

El 01/12/15 a les 17.48, Iurii Mykhalskyi ha escrit:
>> Does something like the following work? If not, could you paste the
>> error when running it with -vvv.
>>
>> xl block-attach DomU
>> format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
> In dom0 we have next issue:
> /libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
> vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
> /this issue occurs due to missing /dev/sda1 device (all hardware are
> placed in DomD domain).

I'm not sure how can you get to this path, the libxl chunk in 
stable-4.5 is:

271     if (disk->format == LIBXL_DISK_FORMAT_EMPTY) {
272         if (!disk->is_cdrom) {
273             LOG(ERROR, "Disk vdev=%s is empty but not cdrom", disk->vdev);
274             return ERROR_INVAL;
275         }
276         memset(&a.stab, 0, sizeof(a.stab));
277     } else if ((disk->backend == LIBXL_DISK_BACKEND_UNKNOWN ||
278                 disk->backend == LIBXL_DISK_BACKEND_PHY) &&
279                disk->backend_domid == LIBXL_TOOLSTACK_DOMID &&
280                !disk->script) {
281         if (stat(disk->pdev_path, &a.stab)) {
282             LOGE(ERROR, "Disk vdev=%s failed to stat: %s",
283                         disk->vdev, disk->pdev_path);
284             return ERROR_INVAL;
285         }
286     }

So it seems that block-attach is ignoring the 'backend=foo' field in 
the disk configuration?

Can you paste the full output of the execution with -vvv?

Roger.

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Doug Goldstein]

[-- Attachment #1.1: Type: text/plain, Size: 4401 bytes --]

On 12/1/15 11:05 AM, Ian Campbell wrote:
> On Tue, 2015-12-01 at 18:48 +0200, Iurii Mykhalskyi wrote:
>> Thanks to all for a replays, please see my answers below:
>>
>> On 12/01/2015 05:29 PM, Wei Liu wrote:
>>> On Tue, Dec 01, 2015 at 04:58:55PM +0200, Iurii Mykhalskyi wrote:
>>>> Our real usb mass-storage device are located at driver domain (DomD).
>>>> So we
>>>> setup second block-device backend there.
>>>>
>>>> To hotplug usb mass-storage from DomD we use follow command:
>>>>
>>>> xl block-attach domU_id phy:/bla-bla,xvda10,w,backend="DomD"
>>>>
>>> What happens if you run this in Dom0? I guess DomD doesn't respond to
>>> the request?
>>     Yes, there is no responded from domD, because actual storage device
>> are located there, and toolstack stuck on real device existence check.
> 
> This check is a toolstack bug which should be fixed. We've squashed some of
> them at various points, but I'd not be surprised if others remain.
> 
>>>> There was no support of attaching block-device in runtime from domain
>>>> other
>>>> to Domain-0, so we have made some hacks to allow call block-attach
>>>> command
>>>> from non-dom0 privileged domain.
>>> So this is a special use case. This is the first time I know people
>>> actually run xl block-attach in driver domain.
>>     Yes, this is special case and we this by our solution design.
>>>> One of patches was - don't update
>>>> /var/lib/xen/userdata-d.$DOMID-$UUID.libxl-json during execution of
>>>> this
>>>> command (because this log located on dom0 rootfs and we don't have
>>>> any
>>>> access to it from DomD). So, there is no different in configs before
>>>> and
>>>> after hotplug.
>>>>
>>> The state of $DOMID is recorded in libxl-json file. No wonder you lose
>>> all state.
>>>
>>> But even if you write those states, they are going to be inside driver
>>> domain.  There is no way at the moment to synthesise the state inside
>>> Dom0 and DomD into one. There is also difficulty in how you can split
>>> the synthesised and dispatch the states to multiple entities again when
>>> rebuilding a domain.
>>>
>>> So I think having multiple entities managing state of one single domain
>>> is bad. I think the proper way of making it work is to make hotplug
>>> device from domain other than Dom0 work.
>>>
>>> There is a daemon "xl devd" in driver domain. We might be able to teach
>>> it to response to Dom0 toostack request. I'm a bit surprised if it
>>> doesn't do that already. Did you forget to start that daemon?
>>     We can't run devd in driver domain, because it failed on connect to
>> xenstored socket (/var/run/xenstored/socket - we have xenstored running
>> only in dom0).   
> 
> devd _should_ be able to talk to xenstored over the kernel provided
> interface to the shared ring rather than the local socket.
> 
> It is certainly not expected that devd be colocated in the same domain as
> happens to be running xenstored.
> 
> If this is not working then there is another bug.

This works, but might have problems in Xen 4.5. If you're using running
on Linux 3.14 or newer then you will have a problem. You need to
backport commit 9c89dc95201ffed5fead17b35754bf9440fdbdc0 if you're using
the C based xenstore and 7d418eab3b6dbdeec84bf73af301dca54368547a if
you're using the Ocaml based xenstore.

You can test this locally by supplying "--disable-socket" to xenstore
when it starts up.


> 
>>> In general a driver domain would not be expected to have sufficient
>>> privilege over e.g. a guest domain's /local/domain/domU/devices to
>>> create
>>> the f.e. dirs.
> 
>>     In our solution we have to create 2 full privileged domains - Dom0 and DomD, so we need 2 toolstack domains.
>>    Any special privileges hack wasn't done - we need just to setup additional permissions for DomD.
> 
> I'm afraid you simply cannot have 2 toolstack domains. The toolstack is a
> singleton entity in a Xen system.
> 
> If you want to run toolstack operations from a non-toolstack domain then
> you will need to arrange for some (likely out-of-band) mechanism for such
> domains to ask the single toolstack domain to do something on their behalf.
> 
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 


-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Ian Campbell]

On Tue, 2015-12-01 at 17:21 +0000, Wei Liu wrote:
> 
> > In dom0 we have next issue:
> > /libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
> > vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
> > /this issue occurs due to missing /dev/sda1 device (all hardware are
> > placed in DomD domain).
> 
> Looks like the path that is stat'ed is not correct.

The path is within the driver domain, while the stat is in the toolstack
domain, so it cannot possibly be correct.

It is a bug if the toolstack is trying to do that stat when toolstack-domid 
!= driver-domain-domid.

Ian.

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Ian Campbell]

On Tue, 2015-12-01 at 18:52 +0100, Roger Pau Monné wrote:
> El 01/12/15 a les 17.48, Iurii Mykhalskyi ha escrit:
> > > Does something like the following work? If not, could you paste the
> > > error when running it with -vvv.
> > > 
> > > xl block-attach DomU
> > > format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
> > In dom0 we have next issue:
> > /libxl: error: libxl_device.c:283:libxl__device_disk_set_backend: Disk
> > vdev=xvda10 failed to stat: /dev/sda1: No such file or directory//-
> > /this issue occurs due to missing /dev/sda1 device (all hardware are
> > placed in DomD domain).
> 
> I'm not sure how can you get to this path, the libxl chunk in 
> stable-4.5 is:
> 
> 271     if (disk->format == LIBXL_DISK_FORMAT_EMPTY) {
> 272         if (!disk->is_cdrom) {
> 273             LOG(ERROR, "Disk vdev=%s is empty but not cdrom", disk-
> >vdev);
> 274             return ERROR_INVAL;
> 275         }
> 276         memset(&a.stab, 0, sizeof(a.stab));
> 277     } else if ((disk->backend == LIBXL_DISK_BACKEND_UNKNOWN ||
> 278                 disk->backend == LIBXL_DISK_BACKEND_PHY) &&
> 279                disk->backend_domid == LIBXL_TOOLSTACK_DOMID &&
> 280                !disk->script) {
> 281         if (stat(disk->pdev_path, &a.stab)) {
> 282             LOGE(ERROR, "Disk vdev=%s failed to stat: %s",
> 283                         disk->vdev, disk->pdev_path);
> 284             return ERROR_INVAL;
> 285         }
> 286     }
> 
> So it seems that block-attach is ignoring the 'backend=foo' field in 
> the disk configuration?
> 
> Can you paste the full output of the execution with -vvv?

Also a dummy attach will print the parsed json of the requested spec, e.g.:

# xl -N block-attach 0 format=raw,vdev=hdc,access=rw,backend=DomD,target=/path/to/dev
disk: {
    "backend_domname": "DomD",
    "pdev_path": "/path/to/dev",
    "vdev": "hdc",
    "format": "raw",
    "readwrite": 1
}

I'm not sure if -vvv on a proper attach will do the same in 4.5, so having
the output of both would be useful.

Ian.
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hotplugged devices in Xen 4.5 and domain reboot

[Ian Campbell]

On Tue, 2015-12-01 at 15:03 -0600, Doug Goldstein wrote:
> This works, but might have problems in Xen 4.5. If you're using running
> on Linux 3.14 or newer then you will have a problem. You need to
> backport commit 9c89dc95201ffed5fead17b35754bf9440fdbdc0 if you're using
> the C based xenstore and 7d418eab3b6dbdeec84bf73af301dca54368547a if
> you're using the Ocaml based xenstore.

What's the underlying problem in Linux which this commit happens to
workaround?

Ian.